2024-03-25T16:24:46.856 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T16:24:46.856 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_amathuri@teuthology --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621281 --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621281/orig.config.yaml 2024-03-25T16:24:46.900 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T16:24:46.998 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621281 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621281' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi003.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGb6LbRgpGpUFYcRAE4NBLu+f/g5s85+pn1GZ0fF1ALVY6cOt3bertjioP6NLkPBZ2n6pGSpneMDUmqNHXIINj0= smithi016.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDZzY+f/rTpKVXTkcqZ4u8ZEZyO6gAkY6E41GQQjYvdPQO9C1pOLTahnFTuRxn6R+lK4BV9WlwaNA0pFgkOC2Gg= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T16:24:46.999 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T16:24:47.000 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T16:24:47.000 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T16:24:47.002 INFO:teuthology.task.internal:Checking packages... 2024-03-25T16:24:47.026 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T16:24:47.026 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T16:24:47.027 INFO:teuthology.packaging:ref: None 2024-03-25T16:24:47.027 INFO:teuthology.packaging:tag: None 2024-03-25T16:24:47.027 INFO:teuthology.packaging:branch: main 2024-03-25T16:24:47.027 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:24:47.027 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T16:24:47.256 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T16:24:47.258 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T16:24:47.264 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T16:24:47.264 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T16:24:47.273 INFO:teuthology.task.internal:Saving configuration 2024-03-25T16:24:47.294 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T16:24:47.301 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T16:24:47.325 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621281', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:19:58.073540', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGb6LbRgpGpUFYcRAE4NBLu+f/g5s85+pn1GZ0fF1ALVY6cOt3bertjioP6NLkPBZ2n6pGSpneMDUmqNHXIINj0='} 2024-03-25T16:24:47.345 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621281', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:19:58.072166', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDZzY+f/rTpKVXTkcqZ4u8ZEZyO6gAkY6E41GQQjYvdPQO9C1pOLTahnFTuRxn6R+lK4BV9WlwaNA0pFgkOC2Gg='} 2024-03-25T16:24:47.346 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T16:24:47.352 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T16:24:47.352 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T16:24:47.352 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T16:24:47.421 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1eb5de10d0>, signals=[15]) 2024-03-25T16:24:47.421 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T16:24:47.428 INFO:teuthology.task.internal:Opening connections... 2024-03-25T16:24:47.428 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi003.front.sepia.ceph.com 2024-03-25T16:24:47.430 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:24:47.534 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi016.front.sepia.ceph.com 2024-03-25T16:24:47.535 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:24:47.637 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T16:24:47.643 DEBUG:teuthology.orchestra.run.smithi003:> uname -m 2024-03-25T16:24:47.661 INFO:teuthology.orchestra.run.smithi003.stdout:x86_64 2024-03-25T16:24:47.661 DEBUG:teuthology.orchestra.run.smithi003:> cat /etc/os-release 2024-03-25T16:24:47.719 INFO:teuthology.orchestra.run.smithi003.stdout:NAME="CentOS Stream" 2024-03-25T16:24:47.720 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION="9" 2024-03-25T16:24:47.720 INFO:teuthology.orchestra.run.smithi003.stdout:ID="centos" 2024-03-25T16:24:47.720 INFO:teuthology.orchestra.run.smithi003.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:24:47.720 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION_ID="9" 2024-03-25T16:24:47.720 INFO:teuthology.orchestra.run.smithi003.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:24:47.720 INFO:teuthology.orchestra.run.smithi003.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:24:47.720 INFO:teuthology.orchestra.run.smithi003.stdout:ANSI_COLOR="0;31" 2024-03-25T16:24:47.720 INFO:teuthology.orchestra.run.smithi003.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:24:47.720 INFO:teuthology.orchestra.run.smithi003.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:24:47.720 INFO:teuthology.orchestra.run.smithi003.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:24:47.720 INFO:teuthology.orchestra.run.smithi003.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:24:47.720 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:24:47.720 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:24:47.721 INFO:teuthology.lock.ops:Updating smithi003.front.sepia.ceph.com on lock server 2024-03-25T16:24:47.743 DEBUG:teuthology.orchestra.run.smithi016:> uname -m 2024-03-25T16:24:47.760 INFO:teuthology.orchestra.run.smithi016.stdout:x86_64 2024-03-25T16:24:47.760 DEBUG:teuthology.orchestra.run.smithi016:> cat /etc/os-release 2024-03-25T16:24:47.817 INFO:teuthology.orchestra.run.smithi016.stdout:NAME="CentOS Stream" 2024-03-25T16:24:47.817 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION="9" 2024-03-25T16:24:47.817 INFO:teuthology.orchestra.run.smithi016.stdout:ID="centos" 2024-03-25T16:24:47.817 INFO:teuthology.orchestra.run.smithi016.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:24:47.817 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_ID="9" 2024-03-25T16:24:47.817 INFO:teuthology.orchestra.run.smithi016.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:24:47.818 INFO:teuthology.orchestra.run.smithi016.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:24:47.818 INFO:teuthology.orchestra.run.smithi016.stdout:ANSI_COLOR="0;31" 2024-03-25T16:24:47.818 INFO:teuthology.orchestra.run.smithi016.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:24:47.818 INFO:teuthology.orchestra.run.smithi016.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:24:47.818 INFO:teuthology.orchestra.run.smithi016.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:24:47.818 INFO:teuthology.orchestra.run.smithi016.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:24:47.818 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:24:47.818 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:24:47.818 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server 2024-03-25T16:24:47.843 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T16:24:47.855 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T16:24:47.860 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T16:24:47.861 DEBUG:teuthology.orchestra.run.smithi003:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:24:47.863 DEBUG:teuthology.orchestra.run.smithi016:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:24:47.878 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T16:24:47.884 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T16:24:47.885 DEBUG:teuthology.orchestra.run.smithi003:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:24:47.921 DEBUG:teuthology.orchestra.run.smithi016:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:24:47.944 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T16:24:47.998 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T16:24:48.015 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T16:24:48.015 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T16:24:48.015 DEBUG:teuthology.orchestra.run.smithi003:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:24:48.016 DEBUG:teuthology.orchestra.run.smithi016:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:24:48.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:24:48.032 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-03-25T16:24:48.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:24:48.034 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2024-03-25T16:24:48.090 INFO:teuthology.orchestra.run.smithi016.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:24:48.090 INFO:teuthology.task.kernel:Running kernel on smithi016: 5.14.0-361.el9.x86_64 2024-03-25T16:24:48.090 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum install -y kernel 2024-03-25T16:24:48.091 INFO:teuthology.orchestra.run.smithi003.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:24:48.092 INFO:teuthology.task.kernel:Running kernel on smithi003: 5.14.0-361.el9.x86_64 2024-03-25T16:24:48.092 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum install -y kernel 2024-03-25T16:24:50.103 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-25T16:24:50.265 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-25T16:24:51.537 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-25T16:24:51.682 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.0 MB 00:00 2024-03-25T16:24:52.536 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-25T16:24:52.540 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-25T16:24:54.237 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - AppStream 30 MB/s | 19 MB 00:00 2024-03-25T16:24:56.649 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - AppStream 6.2 MB/s | 19 MB 00:03 2024-03-25T16:24:59.529 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.1 MB 00:01 2024-03-25T16:25:00.866 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - CRB 9.4 MB/s | 6.1 MB 00:00 2024-03-25T16:25:01.106 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 16 kB 00:00 2024-03-25T16:25:02.157 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-25T16:25:02.234 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - Extras packages 101 kB/s | 16 kB 00:00 2024-03-25T16:25:04.101 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-25T16:25:07.331 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-25T16:25:09.343 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 21 kB/s | 1.7 kB 00:00 2024-03-25T16:25:09.539 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:25:09.539 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:25:09.628 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T16:25:09.637 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:25:09.637 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-03-25T16:25:09.637 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:25:09.638 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T16:25:09.638 INFO:teuthology.orchestra.run.smithi016.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:25:09.638 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-03-25T16:25:09.638 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:25:09.638 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:25:09.638 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:25:09.638 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:25:09.638 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T16:25:09.638 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:25:09.638 INFO:teuthology.orchestra.run.smithi016.stdout:Install 4 Packages 2024-03-25T16:25:09.638 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:25:09.639 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 76 M 2024-03-25T16:25:09.639 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 123 M 2024-03-25T16:25:09.639 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T16:25:09.916 INFO:teuthology.orchestra.run.smithi016.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 423 kB/s | 44 kB 00:00 2024-03-25T16:25:10.516 INFO:teuthology.orchestra.run.smithi016.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 21 MB/s | 15 MB 00:00 2024-03-25T16:25:11.075 INFO:teuthology.orchestra.run.smithi016.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 26 MB/s | 33 MB 00:01 2024-03-25T16:25:11.534 INFO:teuthology.orchestra.run.smithi016.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 17 MB/s | 27 MB 00:01 2024-03-25T16:25:11.534 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:11.534 INFO:teuthology.orchestra.run.smithi016.stdout:Total 40 MB/s | 76 MB 00:01 2024-03-25T16:25:11.558 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:25:11.558 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:25:11.658 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T16:25:11.666 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:25:11.666 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2024-03-25T16:25:11.666 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:25:11.666 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T16:25:11.666 INFO:teuthology.orchestra.run.smithi003.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:25:11.666 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-25T16:25:11.666 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:25:11.667 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:25:11.667 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:25:11.667 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:25:11.667 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T16:25:11.667 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:25:11.667 INFO:teuthology.orchestra.run.smithi003.stdout:Install 4 Packages 2024-03-25T16:25:11.667 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:25:11.668 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 76 M 2024-03-25T16:25:11.668 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 123 M 2024-03-25T16:25:11.668 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T16:25:11.819 INFO:teuthology.orchestra.run.smithi003.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 505 kB/s | 44 kB 00:00 2024-03-25T16:25:12.153 INFO:teuthology.orchestra.run.smithi003.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 35 MB/s | 15 MB 00:00 2024-03-25T16:25:12.231 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T16:25:12.545 INFO:teuthology.orchestra.run.smithi003.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 38 MB/s | 27 MB 00:00 2024-03-25T16:25:12.660 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T16:25:12.661 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T16:25:12.853 INFO:teuthology.orchestra.run.smithi003.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 30 MB/s | 33 MB 00:01 2024-03-25T16:25:12.854 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:12.854 INFO:teuthology.orchestra.run.smithi003.stdout:Total 64 MB/s | 76 MB 00:01 2024-03-25T16:25:13.548 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T16:25:13.711 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T16:25:13.711 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T16:25:13.970 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T16:25:13.971 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T16:25:15.005 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T16:25:15.005 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T16:25:16.317 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T16:25:16.945 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:25:17.034 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:25:17.613 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T16:25:18.244 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:25:18.330 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:25:19.925 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:25:20.061 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:25:21.220 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:25:21.357 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:25:25.134 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:25:25.180 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:25:26.435 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:25:26.474 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:25:30.283 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:25:31.585 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:26:01.420 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:26:01.421 INFO:teuthology.orchestra.run.smithi016.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:26:01.421 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:26:01.467 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:26:01.900 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:26:01.900 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:26:01.900 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:26:01.900 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:26:02.299 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:26:02.300 INFO:teuthology.orchestra.run.smithi003.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:26:02.300 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:26:02.326 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:26:02.326 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:26:02.326 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T16:26:02.326 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:26:02.326 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:26:02.326 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:26:02.326 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:26:02.326 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:26:02.326 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T16:26:02.346 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:26:02.662 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:26:02.725 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:26:02.725 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:26:02.725 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:26:02.725 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:26:02.801 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:26:02.801 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:26:02.801 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:26:02.801 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:26:02.801 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:26:02.802 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T16:26:02.802 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi016.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:26:02.802 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:26:02.885 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:26:02.885 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:26:02.885 DEBUG:teuthology.orchestra.run.smithi016:> sudo rpm -qi grub2-tools 2024-03-25T16:26:02.922 INFO:teuthology.orchestra.run.smithi016.stdout:Name : grub2-tools 2024-03-25T16:26:02.923 INFO:teuthology.orchestra.run.smithi016.stdout:Epoch : 1 2024-03-25T16:26:02.923 INFO:teuthology.orchestra.run.smithi016.stdout:Version : 2.06 2024-03-25T16:26:02.923 INFO:teuthology.orchestra.run.smithi016.stdout:Release : 68.el9 2024-03-25T16:26:02.923 INFO:teuthology.orchestra.run.smithi016.stdout:Architecture: x86_64 2024-03-25T16:26:02.923 INFO:teuthology.orchestra.run.smithi016.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:26:02.923 INFO:teuthology.orchestra.run.smithi016.stdout:Group : Unspecified 2024-03-25T16:26:02.923 INFO:teuthology.orchestra.run.smithi016.stdout:Size : 8264254 2024-03-25T16:26:02.923 INFO:teuthology.orchestra.run.smithi016.stdout:License : GPLv3+ 2024-03-25T16:26:02.923 INFO:teuthology.orchestra.run.smithi016.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:26:02.923 INFO:teuthology.orchestra.run.smithi016.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:26:02.923 INFO:teuthology.orchestra.run.smithi016.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:26:02.923 INFO:teuthology.orchestra.run.smithi016.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:26:02.923 INFO:teuthology.orchestra.run.smithi016.stdout:Packager : builder@centos.org 2024-03-25T16:26:02.923 INFO:teuthology.orchestra.run.smithi016.stdout:Vendor : CentOS 2024-03-25T16:26:02.923 INFO:teuthology.orchestra.run.smithi016.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:26:02.924 INFO:teuthology.orchestra.run.smithi016.stdout:Summary : Support tools for GRUB. 2024-03-25T16:26:02.924 INFO:teuthology.orchestra.run.smithi016.stdout:Description : 2024-03-25T16:26:02.924 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:26:02.924 INFO:teuthology.orchestra.run.smithi016.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:26:02.924 INFO:teuthology.orchestra.run.smithi016.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:26:02.924 INFO:teuthology.orchestra.run.smithi016.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:26:02.924 INFO:teuthology.orchestra.run.smithi016.stdout:hardware devices. 2024-03-25T16:26:02.924 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:26:02.924 INFO:teuthology.orchestra.run.smithi016.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:26:02.925 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:26:02.925 INFO:teuthology.task.kernel:Updating grub on smithi016 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:26:02.925 DEBUG:teuthology.orchestra.run.smithi016:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:26:03.127 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:26:03.127 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:26:03.127 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T16:26:03.127 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:26:03.127 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:26:03.127 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:26:03.127 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:26:03.127 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:26:03.127 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T16:26:03.483 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:26:03.591 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:26:03.591 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:26:03.591 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:26:03.591 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:26:03.591 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:26:03.591 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T16:26:03.591 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi003.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:26:03.591 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:26:03.603 INFO:teuthology.orchestra.run.smithi016.stderr:Generating grub configuration file ... 2024-03-25T16:26:03.673 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:26:03.673 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:26:03.673 DEBUG:teuthology.orchestra.run.smithi003:> sudo rpm -qi grub2-tools 2024-03-25T16:26:03.711 INFO:teuthology.orchestra.run.smithi003.stdout:Name : grub2-tools 2024-03-25T16:26:03.711 INFO:teuthology.orchestra.run.smithi003.stdout:Epoch : 1 2024-03-25T16:26:03.711 INFO:teuthology.orchestra.run.smithi003.stdout:Version : 2.06 2024-03-25T16:26:03.712 INFO:teuthology.orchestra.run.smithi003.stdout:Release : 68.el9 2024-03-25T16:26:03.712 INFO:teuthology.orchestra.run.smithi003.stdout:Architecture: x86_64 2024-03-25T16:26:03.712 INFO:teuthology.orchestra.run.smithi003.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:26:03.712 INFO:teuthology.orchestra.run.smithi003.stdout:Group : Unspecified 2024-03-25T16:26:03.712 INFO:teuthology.orchestra.run.smithi003.stdout:Size : 8264254 2024-03-25T16:26:03.712 INFO:teuthology.orchestra.run.smithi003.stdout:License : GPLv3+ 2024-03-25T16:26:03.712 INFO:teuthology.orchestra.run.smithi003.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:26:03.712 INFO:teuthology.orchestra.run.smithi003.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:26:03.712 INFO:teuthology.orchestra.run.smithi003.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:26:03.712 INFO:teuthology.orchestra.run.smithi003.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:26:03.712 INFO:teuthology.orchestra.run.smithi003.stdout:Packager : builder@centos.org 2024-03-25T16:26:03.712 INFO:teuthology.orchestra.run.smithi003.stdout:Vendor : CentOS 2024-03-25T16:26:03.712 INFO:teuthology.orchestra.run.smithi003.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:26:03.713 INFO:teuthology.orchestra.run.smithi003.stdout:Summary : Support tools for GRUB. 2024-03-25T16:26:03.713 INFO:teuthology.orchestra.run.smithi003.stdout:Description : 2024-03-25T16:26:03.713 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:26:03.713 INFO:teuthology.orchestra.run.smithi003.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:26:03.713 INFO:teuthology.orchestra.run.smithi003.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:26:03.713 INFO:teuthology.orchestra.run.smithi003.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:26:03.713 INFO:teuthology.orchestra.run.smithi003.stdout:hardware devices. 2024-03-25T16:26:03.713 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:26:03.713 INFO:teuthology.orchestra.run.smithi003.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:26:03.714 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:26:03.714 INFO:teuthology.task.kernel:Updating grub on smithi003 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:26:03.714 DEBUG:teuthology.orchestra.run.smithi003:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:26:04.353 INFO:teuthology.orchestra.run.smithi003.stderr:Generating grub configuration file ... 2024-03-25T16:26:05.061 INFO:teuthology.orchestra.run.smithi016.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:26:05.109 INFO:teuthology.orchestra.run.smithi016.stderr:done 2024-03-25T16:26:05.112 DEBUG:teuthology.orchestra.run.smithi016:> mktemp 2024-03-25T16:26:05.128 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.nVhfQ7lD6f 2024-03-25T16:26:05.128 DEBUG:teuthology.orchestra.run.smithi016:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.nVhfQ7lD6f 2024-03-25T16:26:05.201 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.nVhfQ7lD6f 2024-03-25T16:26:05.346 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.nVhfQ7lD6f is 6KB 2024-03-25T16:26:05.396 DEBUG:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.nVhfQ7lD6f 2024-03-25T16:26:05.411 DEBUG:teuthology.orchestra.run.smithi016:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:26:05.480 INFO:teuthology.orchestra.run.smithi016.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:26:05.480 INFO:teuthology.orchestra.run.smithi016.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:26:05.480 INFO:teuthology.orchestra.run.smithi016.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:26:05.480 INFO:teuthology.orchestra.run.smithi016.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:26:05.481 DEBUG:teuthology.orchestra.run.smithi016:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:26:05.620 DEBUG:teuthology.orchestra.run.smithi016:> sudo shutdown -r now 2024-03-25T16:26:05.694 INFO:teuthology.orchestra.run.smithi003.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:26:05.762 INFO:teuthology.orchestra.run.smithi003.stderr:done 2024-03-25T16:26:05.764 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2024-03-25T16:26:05.779 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.e7bxegGQrY 2024-03-25T16:26:05.779 DEBUG:teuthology.orchestra.run.smithi003:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.e7bxegGQrY 2024-03-25T16:26:05.859 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.e7bxegGQrY 2024-03-25T16:26:06.006 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.e7bxegGQrY is 6KB 2024-03-25T16:26:06.055 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.e7bxegGQrY 2024-03-25T16:26:06.089 DEBUG:teuthology.orchestra.run.smithi003:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:26:06.137 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:26:06.138 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:26:06.138 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:26:06.138 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:26:06.139 DEBUG:teuthology.orchestra.run.smithi003:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:26:06.271 DEBUG:teuthology.orchestra.run.smithi003:> sudo shutdown -r now 2024-03-25T16:26:35.650 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T16:26:35.651 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-03-25T16:26:35.652 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:26:36.274 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T16:26:36.274 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-25T16:26:36.275 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:26:54.110 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.16 2024-03-25T16:26:54.626 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.3 2024-03-25T16:27:03.120 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-03-25T16:27:03.121 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:27:03.628 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-25T16:27:03.629 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:28:03.184 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T16:28:03.631 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T16:28:15.197 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-03-25T16:28:15.198 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:28:15.461 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-03-25T16:28:15.633 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-25T16:28:15.633 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:28:15.868 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-03-25T16:28:15.868 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:28:15.869 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-03-25T16:28:15.884 INFO:teuthology.orchestra.run.smithi016.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:28:15.884 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:28:15.885 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:28:15.885 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:28:15.893 DEBUG:teuthology.orchestra.run.smithi003:> true 2024-03-25T16:28:16.276 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-25T16:28:16.276 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:28:16.276 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2024-03-25T16:28:16.291 INFO:teuthology.orchestra.run.smithi003.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:28:16.292 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:28:16.292 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:28:16.292 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:28:16.886 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T16:28:16.886 DEBUG:teuthology.orchestra.run.smithi016:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:28:16.981 INFO:teuthology.orchestra.run.smithi016.stdout:ttyS1 2024-03-25T16:28:17.014 DEBUG:teuthology.parallel:result is None 2024-03-25T16:28:17.293 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T16:28:17.293 DEBUG:teuthology.orchestra.run.smithi003:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:28:17.357 INFO:teuthology.orchestra.run.smithi003.stdout:ttyS1 2024-03-25T16:28:17.390 DEBUG:teuthology.parallel:result is None 2024-03-25T16:28:17.390 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T16:28:17.397 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T16:28:17.397 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:28:17.399 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:28:17.417 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T16:28:17.423 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T16:28:17.432 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T16:28:17.433 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:28:17.456 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:28:17.484 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T16:28:17.490 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T16:28:17.491 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:28:17.516 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:28:17.545 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:28:17.558 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:28:17.559 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:28:17.573 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:28:17.575 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T16:28:17.582 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T16:28:17.582 DEBUG:teuthology.orchestra.run.smithi003:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:28:17.603 DEBUG:teuthology.orchestra.run.smithi016:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:28:17.644 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T16:28:17.656 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T16:28:17.656 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:28:17.672 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:28:17.699 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:28:17.766 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:28:17.814 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:28:17.891 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:28:17.937 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-25T16:28:17.938 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:28:18.004 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:28:18.047 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:28:18.103 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:28:18.187 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:28:18.231 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-25T16:28:18.232 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:28:18.299 DEBUG:teuthology.orchestra.run.smithi003:> sudo service rsyslog restart 2024-03-25T16:28:18.302 DEBUG:teuthology.orchestra.run.smithi016:> sudo service rsyslog restart 2024-03-25T16:28:18.341 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:28:18.383 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:28:18.865 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T16:28:18.877 INFO:teuthology.task.internal:Starting timer... 2024-03-25T16:28:18.877 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T16:28:18.888 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T16:28:18.918 DEBUG:teuthology.orchestra.run.smithi003:> sudo service auditd rotate 2024-03-25T16:28:18.971 INFO:teuthology.orchestra.run.smithi003.stdout:Rotating logs: 2024-03-25T16:28:18.973 DEBUG:teuthology.orchestra.run.smithi016:> sudo service auditd rotate 2024-03-25T16:28:19.027 INFO:teuthology.orchestra.run.smithi016.stdout:Rotating logs: 2024-03-25T16:28:19.029 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T16:28:19.029 DEBUG:teuthology.orchestra.run.smithi003:> /usr/sbin/getenforce 2024-03-25T16:28:19.053 INFO:teuthology.orchestra.run.smithi003.stdout:Permissive 2024-03-25T16:28:19.053 DEBUG:teuthology.orchestra.run.smithi016:> /usr/sbin/getenforce 2024-03-25T16:28:19.081 INFO:teuthology.orchestra.run.smithi016.stdout:Permissive 2024-03-25T16:28:19.081 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi003.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi016.front.sepia.ceph.com': 'permissive'} 2024-03-25T16:28:19.081 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:28:19.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:28:19.122 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:28:19.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:28:19.150 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T16:28:19.150 DEBUG:teuthology.orchestra.run.smithi003:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:28:19.206 DEBUG:teuthology.orchestra.run.smithi016:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:28:19.258 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T16:28:19.267 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T16:28:19.286 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T16:28:19.288 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi003.front.sepia.ceph.com,smithi016.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T16:32:38.180 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi003.front.sepia.ceph.com'), Remote(name='ubuntu@smithi016.front.sepia.ceph.com')] 2024-03-25T16:32:38.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-25T16:32:38.183 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:32:38.273 DEBUG:teuthology.orchestra.run.smithi003:> true 2024-03-25T16:32:38.350 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-25T16:32:38.350 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-03-25T16:32:38.351 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:32:38.441 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-03-25T16:32:38.517 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-03-25T16:32:38.517 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T16:32:38.528 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T16:32:38.529 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:32:38.529 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:32:38.531 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:32:38.532 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:32:38.562 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:32:38.579 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:32:38.589 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:32:38.606 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:32:38.609 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: ntpd: command not found 2024-03-25T16:32:38.623 INFO:teuthology.orchestra.run.smithi003.stdout:506 Cannot talk to daemon 2024-03-25T16:32:38.637 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: ntpd: command not found 2024-03-25T16:32:38.640 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:32:38.650 INFO:teuthology.orchestra.run.smithi016.stdout:506 Cannot talk to daemon 2024-03-25T16:32:38.656 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:32:38.667 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:32:38.683 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:32:38.709 INFO:teuthology.orchestra.run.smithi003.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:32:38.712 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:32:38.712 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2024-03-25T16:32:38.713 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:38.713 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:38.713 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:38.713 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:38.736 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:32:38.740 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:32:38.740 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-03-25T16:32:38.740 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:38.740 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:38.740 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:38.740 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:38.741 INFO:teuthology.run_tasks:Running task install... 2024-03-25T16:32:38.749 DEBUG:teuthology.task.install:project ceph 2024-03-25T16:32:38.749 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T16:32:38.749 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T16:32:38.749 INFO:teuthology.task.install:Using flavor: default 2024-03-25T16:32:38.755 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T16:32:38.755 INFO:teuthology.task.install:extra packages: [] 2024-03-25T16:32:38.756 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:32:38.756 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:32:38.758 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:32:38.758 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:32:38.901 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:32:38.901 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:32:39.037 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:32:39.037 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:32:39.383 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:32:39.383 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-25T16:32:39.383 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:32:39.413 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:32:39.414 DEBUG:teuthology.orchestra.run.smithi016:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:32:39.484 DEBUG:teuthology.orchestra.run.smithi016:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:32:39.503 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:32:39.503 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-25T16:32:39.503 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:32:39.535 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:32:39.535 DEBUG:teuthology.orchestra.run.smithi003:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:32:39.566 DEBUG:teuthology.orchestra.run.smithi016:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:32:39.607 DEBUG:teuthology.orchestra.run.smithi003:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:32:39.634 INFO:teuthology.orchestra.run.smithi016.stdout:check_obsoletes = 1 2024-03-25T16:32:39.636 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum clean all 2024-03-25T16:32:39.692 DEBUG:teuthology.orchestra.run.smithi003:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:32:39.721 INFO:teuthology.orchestra.run.smithi003.stdout:check_obsoletes = 1 2024-03-25T16:32:39.722 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum clean all 2024-03-25T16:32:39.879 INFO:teuthology.orchestra.run.smithi016.stdout:74 files removed 2024-03-25T16:32:39.906 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-radosgw 2024-03-25T16:32:39.975 INFO:teuthology.orchestra.run.smithi003.stdout:74 files removed 2024-03-25T16:32:40.002 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-radosgw 2024-03-25T16:32:41.168 INFO:teuthology.orchestra.run.smithi016.stdout:ceph packages for x86_64 86 kB/s | 82 kB 00:00 2024-03-25T16:32:41.364 INFO:teuthology.orchestra.run.smithi003.stdout:ceph packages for x86_64 77 kB/s | 82 kB 00:01 2024-03-25T16:32:42.085 INFO:teuthology.orchestra.run.smithi016.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T16:32:42.265 INFO:teuthology.orchestra.run.smithi003.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T16:32:43.027 INFO:teuthology.orchestra.run.smithi016.stdout:ceph source packages 2.1 kB/s | 1.9 kB 00:00 2024-03-25T16:32:43.107 INFO:teuthology.orchestra.run.smithi003.stdout:ceph source packages 2.4 kB/s | 1.9 kB 00:00 2024-03-25T16:32:43.563 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-25T16:32:43.976 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - AppStream 8.1 MB/s | 6.7 MB 00:00 2024-03-25T16:32:45.038 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-03-25T16:32:45.451 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-03-25T16:32:47.000 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - BaseOS 6.7 MB/s | 8.0 MB 00:01 2024-03-25T16:32:48.245 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - BaseOS 2.8 MB/s | 8.0 MB 00:02 2024-03-25T16:32:48.967 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-03-25T16:32:54.019 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 6.1 MB 00:01 2024-03-25T16:32:55.686 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-03-25T16:32:57.071 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-25T16:33:02.238 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-25T16:33:04.820 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T16:33:04.834 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:33:04.834 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-25T16:33:04.834 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:33:04.834 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T16:33:04.834 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:33:04.834 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrading: 2024-03-25T16:33:04.834 INFO:teuthology.orchestra.run.smithi003.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:33:04.834 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:33:04.834 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-25T16:33:04.834 INFO:teuthology.orchestra.run.smithi003.stdout: boost-program-options 2024-03-25T16:33:04.834 INFO:teuthology.orchestra.run.smithi003.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:33:04.835 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:33:04.835 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:33:04.835 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:33:04.835 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:33:04.835 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:33:04.835 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:33:04.835 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:33:04.835 INFO:teuthology.orchestra.run.smithi003.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:33:04.835 INFO:teuthology.orchestra.run.smithi003.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:33:04.835 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:33:04.835 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:33:04.835 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:33:04.835 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:33:04.836 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:33:04.836 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:33:04.836 INFO:teuthology.orchestra.run.smithi003.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:33:04.836 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse 2024-03-25T16:33:04.836 INFO:teuthology.orchestra.run.smithi003.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:33:04.836 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:33:04.836 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:33:04.836 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:33:04.836 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:33:04.836 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:33:04.836 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:33:04.836 INFO:teuthology.orchestra.run.smithi003.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:33:04.836 INFO:teuthology.orchestra.run.smithi003.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:33:04.837 INFO:teuthology.orchestra.run.smithi003.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:33:04.837 INFO:teuthology.orchestra.run.smithi003.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:33:04.837 INFO:teuthology.orchestra.run.smithi003.stdout:Installing weak dependencies: 2024-03-25T16:33:04.837 INFO:teuthology.orchestra.run.smithi003.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:33:04.837 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:33:04.837 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T16:33:04.837 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:33:04.837 INFO:teuthology.orchestra.run.smithi003.stdout:Install 30 Packages 2024-03-25T16:33:04.837 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrade 2 Packages 2024-03-25T16:33:04.837 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:33:04.838 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 58 M 2024-03-25T16:33:04.838 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T16:33:05.811 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - AppStream 1.1 MB/s | 19 MB 00:16 2024-03-25T16:33:09.871 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - CRB 18 MB/s | 6.1 MB 00:00 2024-03-25T16:33:11.305 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - Extras packages 70 kB/s | 16 kB 00:00 2024-03-25T16:33:13.923 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 8.3 MB/s | 21 MB 00:02 2024-03-25T16:33:14.771 INFO:teuthology.orchestra.run.smithi003.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 557 kB/s | 5.3 MB 00:09 2024-03-25T16:33:14.913 INFO:teuthology.orchestra.run.smithi003.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 178 kB/s | 25 kB 00:00 2024-03-25T16:33:18.063 INFO:teuthology.orchestra.run.smithi003.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 235 kB/s | 742 kB 00:03 2024-03-25T16:33:19.057 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-03-25T16:33:19.504 INFO:teuthology.orchestra.run.smithi003.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 346 kB/s | 498 kB 00:01 2024-03-25T16:33:21.613 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T16:33:21.617 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:33:21.617 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-03-25T16:33:21.617 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:33:21.617 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T16:33:21.617 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:33:21.617 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrading: 2024-03-25T16:33:21.617 INFO:teuthology.orchestra.run.smithi016.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:33:21.617 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:33:21.617 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-03-25T16:33:21.617 INFO:teuthology.orchestra.run.smithi016.stdout: boost-program-options 2024-03-25T16:33:21.617 INFO:teuthology.orchestra.run.smithi016.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:33:21.617 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:33:21.617 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:33:21.617 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:33:21.617 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:33:21.618 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:33:21.618 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:33:21.618 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:33:21.618 INFO:teuthology.orchestra.run.smithi016.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:33:21.618 INFO:teuthology.orchestra.run.smithi016.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:33:21.618 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:33:21.618 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:33:21.618 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:33:21.618 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:33:21.618 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:33:21.618 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:33:21.618 INFO:teuthology.orchestra.run.smithi016.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:33:21.618 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse 2024-03-25T16:33:21.619 INFO:teuthology.orchestra.run.smithi016.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:33:21.619 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:33:21.619 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:33:21.619 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:33:21.619 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:33:21.619 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:33:21.619 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:33:21.619 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:33:21.619 INFO:teuthology.orchestra.run.smithi016.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:33:21.619 INFO:teuthology.orchestra.run.smithi016.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:33:21.619 INFO:teuthology.orchestra.run.smithi016.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:33:21.619 INFO:teuthology.orchestra.run.smithi016.stdout:Installing weak dependencies: 2024-03-25T16:33:21.619 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:33:21.619 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:33:21.620 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T16:33:21.620 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:33:21.620 INFO:teuthology.orchestra.run.smithi016.stdout:Install 30 Packages 2024-03-25T16:33:21.620 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrade 2 Packages 2024-03-25T16:33:21.620 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:33:21.620 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 58 M 2024-03-25T16:33:21.621 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T16:33:31.313 INFO:teuthology.orchestra.run.smithi003.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 442 kB/s | 5.1 MB 00:11 2024-03-25T16:33:31.887 INFO:teuthology.orchestra.run.smithi003.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 78 kB/s | 45 kB 00:00 2024-03-25T16:33:32.304 INFO:teuthology.orchestra.run.smithi003.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 315 kB/s | 131 kB 00:00 2024-03-25T16:33:33.155 INFO:teuthology.orchestra.run.smithi003.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 190 kB/s | 162 kB 00:00 2024-03-25T16:33:34.030 INFO:teuthology.orchestra.run.smithi003.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 366 kB/s | 320 kB 00:00 2024-03-25T16:33:34.763 INFO:teuthology.orchestra.run.smithi003.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 406 kB/s | 298 kB 00:00 2024-03-25T16:33:35.496 INFO:teuthology.orchestra.run.smithi003.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 135 kB/s | 99 kB 00:00 2024-03-25T16:33:35.613 INFO:teuthology.orchestra.run.smithi003.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 911 kB/s | 106 kB 00:00 2024-03-25T16:33:35.656 INFO:teuthology.orchestra.run.smithi003.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-25T16:33:35.689 INFO:teuthology.orchestra.run.smithi003.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T16:33:35.756 INFO:teuthology.orchestra.run.smithi003.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-25T16:33:35.798 INFO:teuthology.orchestra.run.smithi003.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T16:33:35.840 INFO:teuthology.orchestra.run.smithi003.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 991 kB/s | 41 kB 00:00 2024-03-25T16:33:36.052 INFO:teuthology.orchestra.run.smithi003.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 1.0 MB/s | 216 kB 00:00 2024-03-25T16:33:36.190 INFO:teuthology.orchestra.run.smithi003.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 253 kB/s | 35 kB 00:00 2024-03-25T16:33:36.232 INFO:teuthology.orchestra.run.smithi003.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-25T16:33:36.541 INFO:teuthology.orchestra.run.smithi003.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.0 MB/s | 309 kB 00:00 2024-03-25T16:33:37.092 INFO:teuthology.orchestra.run.smithi003.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 8.0 MB/s | 4.4 MB 00:00 2024-03-25T16:33:37.192 INFO:teuthology.orchestra.run.smithi003.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-03-25T16:33:37.250 INFO:teuthology.orchestra.run.smithi003.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 833 kB/s | 49 kB 00:00 2024-03-25T16:33:37.475 INFO:teuthology.orchestra.run.smithi003.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 299 kB/s | 67 kB 00:00 2024-03-25T16:33:37.676 INFO:teuthology.orchestra.run.smithi003.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.1 MB/s | 838 kB 00:00 2024-03-25T16:33:37.801 INFO:teuthology.orchestra.run.smithi003.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-03-25T16:33:37.927 INFO:teuthology.orchestra.run.smithi003.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-03-25T16:33:44.185 INFO:teuthology.orchestra.run.smithi003.stdout:(29/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 254 kB/s | 9.7 MB 00:39 2024-03-25T16:33:44.456 INFO:teuthology.orchestra.run.smithi016.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 243 kB/s | 5.3 MB 00:22 2024-03-25T16:33:44.581 INFO:teuthology.orchestra.run.smithi016.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 201 kB/s | 25 kB 00:00 2024-03-25T16:33:46.448 INFO:teuthology.orchestra.run.smithi016.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 397 kB/s | 742 kB 00:01 2024-03-25T16:33:47.059 INFO:teuthology.orchestra.run.smithi003.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 378 kB/s | 3.4 MB 00:09 2024-03-25T16:33:47.124 INFO:teuthology.orchestra.run.smithi016.stdout:(4/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 396 kB/s | 9.7 MB 00:25 2024-03-25T16:33:48.049 INFO:teuthology.orchestra.run.smithi016.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 311 kB/s | 498 kB 00:01 2024-03-25T16:33:48.307 INFO:teuthology.orchestra.run.smithi016.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 175 kB/s | 45 kB 00:00 2024-03-25T16:33:48.774 INFO:teuthology.orchestra.run.smithi016.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 281 kB/s | 131 kB 00:00 2024-03-25T16:33:49.608 INFO:teuthology.orchestra.run.smithi016.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 194 kB/s | 162 kB 00:00 2024-03-25T16:33:51.225 INFO:teuthology.orchestra.run.smithi016.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 198 kB/s | 320 kB 00:01 2024-03-25T16:33:52.533 INFO:teuthology.orchestra.run.smithi016.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 228 kB/s | 298 kB 00:01 2024-03-25T16:33:52.942 INFO:teuthology.orchestra.run.smithi016.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 243 kB/s | 99 kB 00:00 2024-03-25T16:33:53.100 INFO:teuthology.orchestra.run.smithi016.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 671 kB/s | 106 kB 00:00 2024-03-25T16:33:53.167 INFO:teuthology.orchestra.run.smithi016.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 159 kB 00:00 2024-03-25T16:33:53.217 INFO:teuthology.orchestra.run.smithi016.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 883 kB/s | 44 kB 00:00 2024-03-25T16:33:53.310 INFO:teuthology.orchestra.run.smithi016.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 661 kB 00:00 2024-03-25T16:33:53.359 INFO:teuthology.orchestra.run.smithi016.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-25T16:33:53.402 INFO:teuthology.orchestra.run.smithi016.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 988 kB/s | 41 kB 00:00 2024-03-25T16:33:53.943 INFO:teuthology.orchestra.run.smithi016.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 400 kB/s | 216 kB 00:00 2024-03-25T16:33:54.052 INFO:teuthology.orchestra.run.smithi016.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 322 kB/s | 35 kB 00:00 2024-03-25T16:33:54.152 INFO:teuthology.orchestra.run.smithi016.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 445 kB/s | 44 kB 00:00 2024-03-25T16:33:54.428 INFO:teuthology.orchestra.run.smithi016.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.1 MB/s | 309 kB 00:00 2024-03-25T16:33:54.636 INFO:teuthology.orchestra.run.smithi016.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 21 MB/s | 4.4 MB 00:00 2024-03-25T16:33:54.695 INFO:teuthology.orchestra.run.smithi016.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 446 kB/s | 26 kB 00:00 2024-03-25T16:33:54.754 INFO:teuthology.orchestra.run.smithi016.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 835 kB/s | 49 kB 00:00 2024-03-25T16:33:54.804 INFO:teuthology.orchestra.run.smithi016.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-03-25T16:33:54.921 INFO:teuthology.orchestra.run.smithi016.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 838 kB 00:00 2024-03-25T16:33:55.029 INFO:teuthology.orchestra.run.smithi016.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-03-25T16:33:55.180 INFO:teuthology.orchestra.run.smithi016.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-25T16:33:56.442 INFO:teuthology.orchestra.run.smithi003.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 261 kB/s | 3.1 MB 00:12 2024-03-25T16:34:04.938 INFO:teuthology.orchestra.run.smithi016.stdout:(29/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 293 kB/s | 5.1 MB 00:17 2024-03-25T16:34:06.429 INFO:teuthology.orchestra.run.smithi016.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 307 kB/s | 3.4 MB 00:11 2024-03-25T16:34:11.045 INFO:teuthology.orchestra.run.smithi016.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 525 kB/s | 3.1 MB 00:06 2024-03-25T16:34:12.673 INFO:teuthology.orchestra.run.smithi003.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 301 kB/s | 20 MB 01:07 2024-03-25T16:34:12.676 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:34:12.676 INFO:teuthology.orchestra.run.smithi003.stdout:Total 873 kB/s | 58 MB 01:07 2024-03-25T16:34:12.706 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T16:34:12.746 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T16:34:12.746 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T16:34:13.236 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T16:34:13.237 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T16:34:14.707 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T16:34:14.790 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:34:14.819 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:34:14.845 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:34:14.868 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:34:14.884 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:34:15.070 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:34:15.149 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:34:15.494 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:34:15.538 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:34:15.631 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:34:15.729 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:34:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:34:15.811 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:34:15.840 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:34:15.875 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:34:15.946 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:34:15.967 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:34:16.021 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:34:16.048 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:34:16.160 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:34:16.191 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:34:16.232 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:34:16.256 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:34:16.285 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:34:16.318 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:34:16.662 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:34:16.690 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:34:16.780 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:34:16.811 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:34:16.850 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:34:17.329 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:34:17.359 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:34:17.430 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:34:17.501 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:34:17.528 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:34:17.878 INFO:teuthology.orchestra.run.smithi016.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 364 kB/s | 20 MB 00:55 2024-03-25T16:34:17.883 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:34:17.883 INFO:teuthology.orchestra.run.smithi016.stdout:Total 1.0 MB/s | 58 MB 00:56 2024-03-25T16:34:17.913 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T16:34:17.953 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T16:34:17.954 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T16:34:18.445 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T16:34:18.445 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T16:34:19.746 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:34:19.783 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:34:19.943 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T16:34:20.026 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:34:20.058 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:34:20.088 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:34:20.114 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:34:20.130 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:34:20.318 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:34:20.357 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:34:20.371 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:34:20.394 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:34:20.490 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:34:20.490 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:34:20.490 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:34:20.491 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:34:20.537 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:34:20.716 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:34:20.776 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:34:20.877 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:34:20.967 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:34:20.995 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:34:21.058 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:34:21.086 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:34:21.131 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:34:21.200 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:34:21.249 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:34:21.435 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:34:21.462 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:34:21.573 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:34:21.605 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:34:21.652 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:34:21.686 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:34:21.716 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:34:21.748 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:34:22.073 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:34:22.103 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:34:22.185 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:34:22.216 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:34:22.262 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:34:22.740 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:34:22.772 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:34:22.843 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:34:22.915 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:34:22.949 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:34:25.179 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:34:25.212 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:34:25.790 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:34:25.833 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:34:25.925 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:34:25.926 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:34:25.926 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:34:25.926 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:34:25.959 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:34:30.090 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:34:30.090 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /sys 2024-03-25T16:34:30.090 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /proc 2024-03-25T16:34:30.090 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /mnt 2024-03-25T16:34:30.090 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /var/tmp 2024-03-25T16:34:30.090 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /home 2024-03-25T16:34:30.090 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /root 2024-03-25T16:34:30.091 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /tmp 2024-03-25T16:34:30.091 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:34:31.017 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:34:31.040 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:34:31.040 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:34:31.040 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:34:31.041 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:34:31.041 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:34:31.041 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:34:31.042 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:34:31.140 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:34:31.140 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:34:32.223 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:34:32.223 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:34:32.223 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:34:32.224 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:34:32.224 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:34:32.224 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:34:32.224 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:34:32.224 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:34:32.224 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:34:32.224 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:34:32.224 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:34:32.224 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:34:32.224 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:34:32.224 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:34:32.224 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:34:32.224 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:34:32.224 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:34:32.225 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:34:32.225 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:34:32.225 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:34:32.225 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:34:32.225 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:34:32.225 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:34:32.225 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:34:32.225 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:34:32.225 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:34:32.225 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:34:32.225 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:34:32.225 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:34:32.225 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:34:32.225 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:34:32.226 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:34:32.226 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:34:32.226 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:34:33.340 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:34:33.340 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:34:33.341 INFO:teuthology.orchestra.run.smithi003.stdout:Upgraded: 2024-03-25T16:34:33.341 INFO:teuthology.orchestra.run.smithi003.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:33.341 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:33.341 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T16:34:33.341 INFO:teuthology.orchestra.run.smithi003.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:34:33.341 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:33.341 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:33.341 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:33.341 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:33.341 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:34:33.341 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:34:33.341 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:34:33.341 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:33.342 INFO:teuthology.orchestra.run.smithi003.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:34:33.342 INFO:teuthology.orchestra.run.smithi003.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:34:33.342 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:34:33.342 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:33.342 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:34:33.342 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:33.342 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:34:33.342 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:34:33.342 INFO:teuthology.orchestra.run.smithi003.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:34:33.342 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:33.342 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:33.342 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:33.342 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:34:33.343 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:33.343 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:33.343 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:33.343 INFO:teuthology.orchestra.run.smithi003.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:34:33.343 INFO:teuthology.orchestra.run.smithi003.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:34:33.343 INFO:teuthology.orchestra.run.smithi003.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:34:33.343 INFO:teuthology.orchestra.run.smithi003.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:34:33.343 INFO:teuthology.orchestra.run.smithi003.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:34:33.343 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:34:33.343 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T16:34:33.683 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-test 2024-03-25T16:34:34.252 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:32 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T16:34:34.415 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T16:34:34.416 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:34:34.416 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-25T16:34:34.416 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:34:34.416 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T16:34:34.416 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:34:34.416 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-25T16:34:34.416 INFO:teuthology.orchestra.run.smithi003.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:34:34.416 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:34:34.417 INFO:teuthology.orchestra.run.smithi003.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:34:34.417 INFO:teuthology.orchestra.run.smithi003.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:34:34.417 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:34:34.417 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:34:34.417 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T16:34:34.417 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:34:34.417 INFO:teuthology.orchestra.run.smithi003.stdout:Install 6 Packages 2024-03-25T16:34:34.417 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:34:34.418 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 48 M 2024-03-25T16:34:34.418 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 204 M 2024-03-25T16:34:34.418 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T16:34:34.652 INFO:teuthology.orchestra.run.smithi003.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-25T16:34:34.710 INFO:teuthology.orchestra.run.smithi003.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-25T16:34:34.744 INFO:teuthology.orchestra.run.smithi003.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-25T16:34:34.886 INFO:teuthology.orchestra.run.smithi003.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-25T16:34:35.818 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:34:35.818 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /sys 2024-03-25T16:34:35.818 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /proc 2024-03-25T16:34:35.818 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /mnt 2024-03-25T16:34:35.818 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /var/tmp 2024-03-25T16:34:35.818 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /home 2024-03-25T16:34:35.818 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /root 2024-03-25T16:34:35.819 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /tmp 2024-03-25T16:34:35.819 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:34:36.002 INFO:teuthology.orchestra.run.smithi003.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 108 kB/s | 163 kB 00:01 2024-03-25T16:34:36.748 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:34:36.774 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:34:36.774 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:34:36.774 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:34:36.774 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:34:36.775 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:34:36.775 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:34:36.777 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:34:36.876 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:34:36.876 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:34:38.055 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:34:38.055 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:34:38.055 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:34:38.055 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:34:38.055 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:34:38.055 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:34:38.055 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:34:38.055 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:34:38.055 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:34:38.056 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:34:38.056 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:34:38.056 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:34:38.056 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:34:38.056 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:34:38.056 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:34:38.056 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:34:38.056 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:34:38.056 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:34:38.056 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:34:38.056 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:34:38.056 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:34:38.056 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:34:38.056 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:34:38.057 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:34:38.057 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:34:38.057 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:34:38.057 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:34:38.057 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:34:38.058 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:34:38.058 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:34:38.058 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:34:38.058 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:34:38.058 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:34:38.058 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:34:38.968 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:34:38.968 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:34:38.968 INFO:teuthology.orchestra.run.smithi016.stdout:Upgraded: 2024-03-25T16:34:38.968 INFO:teuthology.orchestra.run.smithi016.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:38.968 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:38.968 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T16:34:38.968 INFO:teuthology.orchestra.run.smithi016.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:34:38.968 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:38.968 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:38.968 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:38.968 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:38.968 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:34:38.968 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:34:38.969 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:34:38.969 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:38.969 INFO:teuthology.orchestra.run.smithi016.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:34:38.969 INFO:teuthology.orchestra.run.smithi016.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:34:38.969 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:34:38.969 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:38.969 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:34:38.969 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:38.969 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:34:38.969 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:34:38.969 INFO:teuthology.orchestra.run.smithi016.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:34:38.969 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:38.969 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:38.969 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:38.969 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:34:38.970 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:38.970 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:38.970 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:38.970 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:34:38.970 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:34:38.970 INFO:teuthology.orchestra.run.smithi016.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:34:38.970 INFO:teuthology.orchestra.run.smithi016.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:34:38.970 INFO:teuthology.orchestra.run.smithi016.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:34:38.970 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:34:38.970 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T16:34:39.287 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-test 2024-03-25T16:34:39.869 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:20 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T16:34:40.028 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T16:34:40.029 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:34:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-03-25T16:34:40.029 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:34:40.029 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T16:34:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:34:40.029 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-03-25T16:34:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:34:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:34:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:34:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:34:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:34:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:34:40.030 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T16:34:40.030 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:34:40.030 INFO:teuthology.orchestra.run.smithi016.stdout:Install 6 Packages 2024-03-25T16:34:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:34:40.031 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 48 M 2024-03-25T16:34:40.031 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 204 M 2024-03-25T16:34:40.031 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T16:34:40.304 INFO:teuthology.orchestra.run.smithi016.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 946 kB/s | 217 kB 00:00 2024-03-25T16:34:40.429 INFO:teuthology.orchestra.run.smithi016.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.3 MB/s | 300 kB 00:00 2024-03-25T16:34:40.496 INFO:teuthology.orchestra.run.smithi016.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 938 kB/s | 63 kB 00:00 2024-03-25T16:34:41.397 INFO:teuthology.orchestra.run.smithi016.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 208 kB/s | 187 kB 00:00 2024-03-25T16:34:41.438 INFO:teuthology.orchestra.run.smithi016.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 120 kB/s | 163 kB 00:01 2024-03-25T16:36:19.399 INFO:teuthology.orchestra.run.smithi003.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 461 kB/s | 47 MB 01:44 2024-03-25T16:36:19.400 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:19.400 INFO:teuthology.orchestra.run.smithi003.stdout:Total 470 kB/s | 48 MB 01:44 2024-03-25T16:36:19.412 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T16:36:19.432 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T16:36:19.432 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T16:36:19.828 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T16:36:19.828 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T16:36:20.454 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T16:36:20.515 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:36:20.555 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:36:20.595 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:36:20.637 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:36:20.658 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:36:25.364 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:36:25.444 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:36:27.920 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:36:27.921 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:36:27.921 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:36:27.921 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:36:27.921 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:36:27.921 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:36:28.273 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:36:28.274 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:36:28.274 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T16:36:28.274 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:28.274 INFO:teuthology.orchestra.run.smithi003.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:36:28.274 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:28.274 INFO:teuthology.orchestra.run.smithi003.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:36:28.274 INFO:teuthology.orchestra.run.smithi003.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:36:28.274 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:36:28.274 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:36:28.274 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T16:36:28.568 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph 2024-03-25T16:36:29.139 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:03:27 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T16:36:29.301 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T16:36:29.306 INFO:teuthology.orchestra.run.smithi003.stdout:========================================================================================= 2024-03-25T16:36:29.306 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-25T16:36:29.307 INFO:teuthology.orchestra.run.smithi003.stdout:========================================================================================= 2024-03-25T16:36:29.307 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T16:36:29.307 INFO:teuthology.orchestra.run.smithi003.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:36:29.307 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-25T16:36:29.307 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:36:29.307 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:36:29.307 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:36:29.307 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:36:29.307 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:36:29.307 INFO:teuthology.orchestra.run.smithi003.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:36:29.307 INFO:teuthology.orchestra.run.smithi003.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:36:29.307 INFO:teuthology.orchestra.run.smithi003.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:36:29.307 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:36:29.308 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:36:29.308 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:36:29.308 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:36:29.308 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:36:29.308 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:36:29.308 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:36:29.308 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:36:29.308 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:36:29.308 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:36:29.308 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:36:29.308 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:36:29.308 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:36:29.308 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:36:29.309 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:36:29.309 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:36:29.309 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:36:29.309 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:36:29.309 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:36:29.309 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:36:29.309 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:36:29.309 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:36:29.309 INFO:teuthology.orchestra.run.smithi003.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:36:29.309 INFO:teuthology.orchestra.run.smithi003.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:36:29.309 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:36:29.309 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:36:29.309 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:36:29.309 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:36:29.310 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T16:36:29.310 INFO:teuthology.orchestra.run.smithi003.stdout:========================================================================================= 2024-03-25T16:36:29.310 INFO:teuthology.orchestra.run.smithi003.stdout:Install 36 Packages 2024-03-25T16:36:29.310 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:36:29.310 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 30 M 2024-03-25T16:36:29.310 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 107 M 2024-03-25T16:36:29.310 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T16:36:30.124 INFO:teuthology.orchestra.run.smithi003.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 15 kB/s | 6.6 kB 00:00 2024-03-25T16:36:35.557 INFO:teuthology.orchestra.run.smithi003.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 260 kB/s | 1.5 MB 00:05 2024-03-25T16:36:37.482 INFO:teuthology.orchestra.run.smithi003.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 279 kB/s | 2.1 MB 00:07 2024-03-25T16:36:38.349 INFO:teuthology.orchestra.run.smithi003.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 285 kB/s | 247 kB 00:00 2024-03-25T16:36:38.483 INFO:teuthology.orchestra.run.smithi003.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-25T16:36:38.550 INFO:teuthology.orchestra.run.smithi003.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.1 MB/s | 144 kB 00:00 2024-03-25T16:36:38.583 INFO:teuthology.orchestra.run.smithi003.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 949 kB/s | 32 kB 00:00 2024-03-25T16:36:38.608 INFO:teuthology.orchestra.run.smithi003.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.9 MB/s | 47 kB 00:00 2024-03-25T16:36:38.634 INFO:teuthology.orchestra.run.smithi003.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-25T16:36:38.684 INFO:teuthology.orchestra.run.smithi003.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-25T16:36:38.834 INFO:teuthology.orchestra.run.smithi003.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 7.6 MB/s | 1.1 MB 00:00 2024-03-25T16:36:38.893 INFO:teuthology.orchestra.run.smithi003.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 1.7 MB/s | 103 kB 00:00 2024-03-25T16:36:38.968 INFO:teuthology.orchestra.run.smithi003.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 1.6 MB/s | 124 kB 00:00 2024-03-25T16:36:39.010 INFO:teuthology.orchestra.run.smithi003.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T16:36:39.210 INFO:teuthology.orchestra.run.smithi003.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 625 kB/s | 125 kB 00:00 2024-03-25T16:36:39.478 INFO:teuthology.orchestra.run.smithi003.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 84 kB/s | 22 kB 00:00 2024-03-25T16:36:39.620 INFO:teuthology.orchestra.run.smithi003.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-03-25T16:36:39.662 INFO:teuthology.orchestra.run.smithi003.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.0 MB/s | 43 kB 00:00 2024-03-25T16:36:39.720 INFO:teuthology.orchestra.run.smithi003.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-03-25T16:36:39.787 INFO:teuthology.orchestra.run.smithi003.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 5.3 MB/s | 358 kB 00:00 2024-03-25T16:36:39.821 INFO:teuthology.orchestra.run.smithi003.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 322 kB/s | 11 kB 00:00 2024-03-25T16:36:39.854 INFO:teuthology.orchestra.run.smithi003.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 532 kB/s | 18 kB 00:00 2024-03-25T16:36:39.888 INFO:teuthology.orchestra.run.smithi003.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 693 kB/s | 23 kB 00:00 2024-03-25T16:36:39.921 INFO:teuthology.orchestra.run.smithi003.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 584 kB/s | 19 kB 00:00 2024-03-25T16:36:39.955 INFO:teuthology.orchestra.run.smithi003.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 591 kB/s | 20 kB 00:00 2024-03-25T16:36:39.997 INFO:teuthology.orchestra.run.smithi003.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2024-03-25T16:36:40.039 INFO:teuthology.orchestra.run.smithi003.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.9 MB/s | 79 kB 00:00 2024-03-25T16:36:40.089 INFO:teuthology.orchestra.run.smithi003.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-25T16:36:40.131 INFO:teuthology.orchestra.run.smithi003.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 395 kB/s | 16 kB 00:00 2024-03-25T16:36:40.173 INFO:teuthology.orchestra.run.smithi003.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-03-25T16:36:40.215 INFO:teuthology.orchestra.run.smithi003.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 862 kB/s | 36 kB 00:00 2024-03-25T16:36:40.265 INFO:teuthology.orchestra.run.smithi003.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-25T16:36:40.324 INFO:teuthology.orchestra.run.smithi003.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 7.2 MB/s | 427 kB 00:00 2024-03-25T16:36:40.365 INFO:teuthology.orchestra.run.smithi003.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 481 kB/s | 20 kB 00:00 2024-03-25T16:36:48.082 INFO:teuthology.orchestra.run.smithi003.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 267 kB/s | 4.7 MB 00:17 2024-03-25T16:37:01.982 INFO:teuthology.orchestra.run.smithi016.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 341 kB/s | 47 MB 02:21 2024-03-25T16:37:01.985 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:01.985 INFO:teuthology.orchestra.run.smithi016.stdout:Total 348 kB/s | 48 MB 02:21 2024-03-25T16:37:01.996 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T16:37:02.017 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T16:37:02.017 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T16:37:02.410 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T16:37:02.411 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T16:37:03.022 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T16:37:03.083 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:37:03.125 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:37:03.165 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:37:03.214 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:37:03.235 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:37:07.992 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:37:08.071 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:37:10.409 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:37:10.409 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:37:10.409 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:37:10.409 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:37:10.409 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:37:10.409 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:37:10.833 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:37:10.834 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:37:10.834 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T16:37:10.834 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:10.834 INFO:teuthology.orchestra.run.smithi016.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:37:10.834 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:10.834 INFO:teuthology.orchestra.run.smithi016.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:37:10.834 INFO:teuthology.orchestra.run.smithi016.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:37:10.834 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:37:10.834 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:37:10.834 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T16:37:11.119 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph 2024-03-25T16:37:11.694 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:03:52 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T16:37:11.855 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T16:37:11.858 INFO:teuthology.orchestra.run.smithi016.stdout:========================================================================================= 2024-03-25T16:37:11.858 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-03-25T16:37:11.858 INFO:teuthology.orchestra.run.smithi016.stdout:========================================================================================= 2024-03-25T16:37:11.859 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T16:37:11.859 INFO:teuthology.orchestra.run.smithi016.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:37:11.859 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-03-25T16:37:11.859 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:37:11.859 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:37:11.859 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:37:11.859 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:37:11.859 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:37:11.859 INFO:teuthology.orchestra.run.smithi016.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:37:11.859 INFO:teuthology.orchestra.run.smithi016.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:37:11.859 INFO:teuthology.orchestra.run.smithi016.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:37:11.859 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:37:11.860 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:37:11.860 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:37:11.860 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:37:11.860 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:37:11.860 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:37:11.860 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:37:11.860 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:37:11.860 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:37:11.860 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:37:11.861 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:37:11.861 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:37:11.861 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:37:11.861 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:37:11.861 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:37:11.861 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:37:11.861 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:37:11.861 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:37:11.861 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:37:11.861 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:37:11.861 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:37:11.861 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:37:11.862 INFO:teuthology.orchestra.run.smithi016.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:37:11.862 INFO:teuthology.orchestra.run.smithi016.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:37:11.862 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:37:11.862 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:37:11.862 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:37:11.862 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:37:11.862 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T16:37:11.862 INFO:teuthology.orchestra.run.smithi016.stdout:========================================================================================= 2024-03-25T16:37:11.862 INFO:teuthology.orchestra.run.smithi016.stdout:Install 36 Packages 2024-03-25T16:37:11.862 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:37:11.863 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 30 M 2024-03-25T16:37:11.863 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 107 M 2024-03-25T16:37:11.863 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T16:37:12.520 INFO:teuthology.orchestra.run.smithi016.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 17 kB/s | 6.6 kB 00:00 2024-03-25T16:37:16.679 INFO:teuthology.orchestra.run.smithi016.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 334 kB/s | 1.5 MB 00:04 2024-03-25T16:37:21.145 INFO:teuthology.orchestra.run.smithi016.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 241 kB/s | 2.1 MB 00:09 2024-03-25T16:37:22.012 INFO:teuthology.orchestra.run.smithi016.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 285 kB/s | 247 kB 00:00 2024-03-25T16:37:22.137 INFO:teuthology.orchestra.run.smithi016.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-25T16:37:22.179 INFO:teuthology.orchestra.run.smithi016.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T16:37:22.213 INFO:teuthology.orchestra.run.smithi016.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 957 kB/s | 32 kB 00:00 2024-03-25T16:37:22.246 INFO:teuthology.orchestra.run.smithi016.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T16:37:22.280 INFO:teuthology.orchestra.run.smithi016.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T16:37:22.331 INFO:teuthology.orchestra.run.smithi016.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-25T16:37:22.423 INFO:teuthology.orchestra.run.smithi016.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 12 MB/s | 1.1 MB 00:00 2024-03-25T16:37:22.465 INFO:teuthology.orchestra.run.smithi016.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-25T16:37:22.506 INFO:teuthology.orchestra.run.smithi016.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-25T16:37:22.548 INFO:teuthology.orchestra.run.smithi016.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T16:37:22.690 INFO:teuthology.orchestra.run.smithi016.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 881 kB/s | 125 kB 00:00 2024-03-25T16:37:22.741 INFO:teuthology.orchestra.run.smithi016.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 446 kB/s | 22 kB 00:00 2024-03-25T16:37:23.174 INFO:teuthology.orchestra.run.smithi016.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 349 kB/s | 151 kB 00:00 2024-03-25T16:37:23.274 INFO:teuthology.orchestra.run.smithi016.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 434 kB/s | 43 kB 00:00 2024-03-25T16:37:23.425 INFO:teuthology.orchestra.run.smithi016.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-03-25T16:37:23.650 INFO:teuthology.orchestra.run.smithi016.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.6 MB/s | 358 kB 00:00 2024-03-25T16:37:23.767 INFO:teuthology.orchestra.run.smithi016.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 92 kB/s | 11 kB 00:00 2024-03-25T16:37:23.876 INFO:teuthology.orchestra.run.smithi016.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 163 kB/s | 18 kB 00:00 2024-03-25T16:37:24.001 INFO:teuthology.orchestra.run.smithi016.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 185 kB/s | 23 kB 00:00 2024-03-25T16:37:24.126 INFO:teuthology.orchestra.run.smithi016.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 155 kB/s | 19 kB 00:00 2024-03-25T16:37:24.252 INFO:teuthology.orchestra.run.smithi016.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 157 kB/s | 20 kB 00:00 2024-03-25T16:37:24.352 INFO:teuthology.orchestra.run.smithi016.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 463 kB/s | 46 kB 00:00 2024-03-25T16:37:24.481 INFO:teuthology.orchestra.run.smithi016.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 613 kB/s | 79 kB 00:00 2024-03-25T16:37:24.602 INFO:teuthology.orchestra.run.smithi016.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.2 MB/s | 272 kB 00:00 2024-03-25T16:37:24.702 INFO:teuthology.orchestra.run.smithi016.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 164 kB/s | 16 kB 00:00 2024-03-25T16:37:24.811 INFO:teuthology.orchestra.run.smithi016.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 829 kB/s | 90 kB 00:00 2024-03-25T16:37:24.911 INFO:teuthology.orchestra.run.smithi016.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 358 kB/s | 36 kB 00:00 2024-03-25T16:37:25.037 INFO:teuthology.orchestra.run.smithi016.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-03-25T16:37:25.179 INFO:teuthology.orchestra.run.smithi016.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.9 MB/s | 427 kB 00:00 2024-03-25T16:37:25.271 INFO:teuthology.orchestra.run.smithi016.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 218 kB/s | 20 kB 00:00 2024-03-25T16:37:27.846 INFO:teuthology.orchestra.run.smithi016.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 313 kB/s | 4.7 MB 00:15 2024-03-25T16:37:28.218 INFO:teuthology.orchestra.run.smithi003.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 326 kB/s | 17 MB 00:52 2024-03-25T16:37:28.221 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:28.221 INFO:teuthology.orchestra.run.smithi003.stdout:Total 516 kB/s | 30 MB 00:58 2024-03-25T16:37:28.245 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T16:37:28.290 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T16:37:28.290 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T16:37:28.712 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T16:37:28.712 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T16:37:30.058 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T16:37:30.162 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:37:30.204 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:37:30.231 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:37:30.305 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:37:30.365 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:37:30.396 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:37:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:37:30.577 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:37:30.624 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:37:30.698 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:37:30.750 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:37:30.788 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:37:30.847 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:37:30.910 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:37:31.105 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:37:31.151 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:37:31.189 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:37:31.232 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:37:31.262 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:37:31.340 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:37:31.381 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:37:31.414 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:37:31.454 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:37:31.490 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:37:31.532 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:37:31.633 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:37:31.679 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:37:31.756 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:37:31.903 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:37:32.034 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:37:32.203 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:37:32.234 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:37:32.259 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:37:32.259 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:37:32.259 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:37:32.260 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:37:32.260 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:37:32.260 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:37:33.785 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:37:33.813 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:37:33.813 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:37:33.813 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:37:33.813 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:37:33.813 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:37:33.813 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:37:34.278 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:37:34.301 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:37:34.301 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:37:34.301 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:37:34.301 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:37:34.301 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:37:34.301 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:37:34.511 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:37:34.535 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:37:34.535 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:37:34.535 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:37:34.535 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:37:34.535 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:37:34.535 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:37:34.595 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:37:36.278 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:37:36.278 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:37:36.279 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:37:36.279 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:37:36.279 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:37:36.279 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:37:36.279 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:37:36.279 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:37:36.279 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:37:36.279 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:37:36.279 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:37:36.279 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:37:36.279 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:37:36.279 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:37:36.279 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:37:36.280 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:37:36.280 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:37:36.280 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:37:36.280 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:37:36.280 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:37:36.280 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:37:36.280 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:37:36.280 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:37:36.280 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:37:36.280 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:37:36.280 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:37:36.280 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:37:36.280 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:37:36.281 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:37:36.282 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:37:36.282 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:37:36.282 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:37:36.282 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:37:36.282 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:37:36.282 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:37:36.282 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:37:37.287 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:37:37.287 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:37:37.287 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T16:37:37.287 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:37.287 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:37.288 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:37.288 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:37:37.288 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:37.288 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:37.288 INFO:teuthology.orchestra.run.smithi003.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:37:37.288 INFO:teuthology.orchestra.run.smithi003.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:37:37.288 INFO:teuthology.orchestra.run.smithi003.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:37:37.288 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:37:37.288 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:37:37.288 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:37:37.288 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:37:37.288 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:37:37.288 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:37:37.289 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:37:37.289 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:37:37.289 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:37:37.289 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:37:37.289 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:37:37.289 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:37:37.289 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:37:37.289 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:37:37.289 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:37:37.289 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:37:37.289 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:37:37.289 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:37:37.289 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:37:37.290 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:37:37.290 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:37:37.290 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:37:37.290 INFO:teuthology.orchestra.run.smithi003.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:37:37.290 INFO:teuthology.orchestra.run.smithi003.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:37:37.290 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:37:37.290 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:37:37.290 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:37:37.290 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:37:37.290 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T16:37:37.639 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-base 2024-03-25T16:37:38.210 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:04:36 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T16:37:38.310 INFO:teuthology.orchestra.run.smithi003.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:37:38.365 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T16:37:38.366 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-25T16:37:38.366 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T16:37:38.435 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install cephadm 2024-03-25T16:37:39.025 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:04:37 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T16:37:39.193 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T16:37:39.194 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:37:39.194 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-25T16:37:39.194 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:37:39.194 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T16:37:39.194 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:37:39.195 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:37:39.195 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T16:37:39.195 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:37:39.195 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-25T16:37:39.195 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:37:39.195 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 747 k 2024-03-25T16:37:39.195 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 754 k 2024-03-25T16:37:39.195 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T16:37:43.641 INFO:teuthology.orchestra.run.smithi003.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 168 kB/s | 747 kB 00:04 2024-03-25T16:37:43.642 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:43.642 INFO:teuthology.orchestra.run.smithi003.stdout:Total 168 kB/s | 747 kB 00:04 2024-03-25T16:37:43.643 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T16:37:43.650 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T16:37:43.650 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T16:37:43.656 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T16:37:43.656 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T16:37:43.734 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T16:37:44.185 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:37:44.284 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:37:44.680 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:37:44.961 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:37:44.961 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:37:44.961 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T16:37:44.961 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:37:44.962 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:37:44.962 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T16:37:45.150 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:37:45.730 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:04:43 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T16:37:45.884 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T16:37:45.885 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:37:45.885 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-25T16:37:45.885 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:37:45.885 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T16:37:45.885 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:37:45.885 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:37:45.885 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T16:37:45.885 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:37:45.885 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-25T16:37:45.885 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:37:45.885 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 145 k 2024-03-25T16:37:45.885 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 436 k 2024-03-25T16:37:45.886 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T16:37:46.972 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 134 kB/s | 145 kB 00:01 2024-03-25T16:37:46.972 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:46.972 INFO:teuthology.orchestra.run.smithi003.stdout:Total 134 kB/s | 145 kB 00:01 2024-03-25T16:37:46.973 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T16:37:46.982 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T16:37:46.982 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T16:37:47.041 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T16:37:47.041 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T16:37:47.194 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T16:37:47.247 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:37:47.272 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:37:47.272 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:37:47.272 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:37:47.272 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:37:48.271 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:37:48.271 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:37:48.272 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T16:37:48.272 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:48.272 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:37:48.272 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T16:37:48.657 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr 2024-03-25T16:37:49.234 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:04:47 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T16:37:49.334 INFO:teuthology.orchestra.run.smithi003.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:37:49.388 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T16:37:49.389 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-25T16:37:49.389 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T16:37:49.466 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:37:50.038 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:04:48 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T16:37:50.194 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T16:37:50.195 INFO:teuthology.orchestra.run.smithi003.stdout:====================================================================================== 2024-03-25T16:37:50.195 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-25T16:37:50.195 INFO:teuthology.orchestra.run.smithi003.stdout:====================================================================================== 2024-03-25T16:37:50.195 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T16:37:50.195 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:37:50.195 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-25T16:37:50.195 INFO:teuthology.orchestra.run.smithi003.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:37:50.196 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:37:50.196 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:37:50.196 INFO:teuthology.orchestra.run.smithi003.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:37:50.196 INFO:teuthology.orchestra.run.smithi003.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:37:50.196 INFO:teuthology.orchestra.run.smithi003.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:37:50.196 INFO:teuthology.orchestra.run.smithi003.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:37:50.196 INFO:teuthology.orchestra.run.smithi003.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:37:50.196 INFO:teuthology.orchestra.run.smithi003.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:37:50.196 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:37:50.196 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:37:50.196 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:37:50.196 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T16:37:50.196 INFO:teuthology.orchestra.run.smithi003.stdout:====================================================================================== 2024-03-25T16:37:50.196 INFO:teuthology.orchestra.run.smithi003.stdout:Install 12 Packages 2024-03-25T16:37:50.197 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:37:50.197 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 6.9 M 2024-03-25T16:37:50.197 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 26 M 2024-03-25T16:37:50.197 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T16:37:51.225 INFO:teuthology.orchestra.run.smithi003.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 56 kB/s | 31 kB 00:00 2024-03-25T16:37:51.342 INFO:teuthology.orchestra.run.smithi003.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 23 kB/s | 15 kB 00:00 2024-03-25T16:37:51.400 INFO:teuthology.orchestra.run.smithi003.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.8 MB/s | 1.0 MB 00:00 2024-03-25T16:37:51.425 INFO:teuthology.orchestra.run.smithi003.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.8 MB/s | 238 kB 00:00 2024-03-25T16:37:51.743 INFO:teuthology.orchestra.run.smithi003.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.5 MB/s | 863 kB 00:00 2024-03-25T16:37:52.018 INFO:teuthology.orchestra.run.smithi003.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 71 kB/s | 19 kB 00:00 2024-03-25T16:37:52.076 INFO:teuthology.orchestra.run.smithi003.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 842 kB/s | 548 kB 00:00 2024-03-25T16:37:52.202 INFO:teuthology.orchestra.run.smithi003.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.1 MB/s | 144 kB 00:00 2024-03-25T16:37:52.327 INFO:teuthology.orchestra.run.smithi003.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 246 kB/s | 31 kB 00:00 2024-03-25T16:37:52.469 INFO:teuthology.orchestra.run.smithi003.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.3 MB/s | 188 kB 00:00 2024-03-25T16:37:52.719 INFO:teuthology.orchestra.run.smithi003.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.9 MB/s | 2.0 MB 00:00 2024-03-25T16:37:56.844 INFO:teuthology.orchestra.run.smithi003.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 295 kB/s | 1.8 MB 00:06 2024-03-25T16:37:56.847 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:56.847 INFO:teuthology.orchestra.run.smithi003.stdout:Total 1.0 MB/s | 6.9 MB 00:06 2024-03-25T16:37:56.867 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T16:37:56.888 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T16:37:56.888 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T16:37:57.079 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T16:37:57.080 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T16:37:57.720 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T16:37:57.787 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:37:57.830 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:37:57.877 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:37:57.915 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:37:58.017 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:37:58.145 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:37:58.438 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:37:58.604 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:37:58.631 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:37:58.663 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:37:58.917 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:37:58.946 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:37:59.624 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:37:59.624 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:37:59.624 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:37:59.624 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:37:59.624 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:37:59.625 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:37:59.625 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:37:59.625 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:37:59.625 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:37:59.625 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:37:59.625 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:37:59.625 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:37:59.961 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:37:59.961 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:37:59.961 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T16:37:59.962 INFO:teuthology.orchestra.run.smithi003.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:37:59.962 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:37:59.962 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:37:59.962 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:37:59.962 INFO:teuthology.orchestra.run.smithi003.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:37:59.962 INFO:teuthology.orchestra.run.smithi003.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:37:59.962 INFO:teuthology.orchestra.run.smithi003.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:37:59.962 INFO:teuthology.orchestra.run.smithi003.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:37:59.962 INFO:teuthology.orchestra.run.smithi003.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:37:59.962 INFO:teuthology.orchestra.run.smithi003.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:37:59.962 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:37:59.962 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:37:59.962 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:37:59.963 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T16:38:00.210 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:38:00.779 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:04:58 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T16:38:00.937 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T16:38:00.938 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================================ 2024-03-25T16:38:00.938 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-25T16:38:00.938 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================================ 2024-03-25T16:38:00.938 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T16:38:00.938 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:38:00.938 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-25T16:38:00.939 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:38:00.939 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:38:00.939 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:38:00.939 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:38:00.939 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:38:00.939 INFO:teuthology.orchestra.run.smithi003.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:38:00.939 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:38:00.940 INFO:teuthology.orchestra.run.smithi003.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:38:00.940 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:38:00.940 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:38:00.940 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:38:00.940 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:38:00.940 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T16:38:00.940 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================================ 2024-03-25T16:38:00.940 INFO:teuthology.orchestra.run.smithi003.stdout:Install 12 Packages 2024-03-25T16:38:00.941 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:38:00.941 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 37 M 2024-03-25T16:38:00.941 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 209 M 2024-03-25T16:38:00.942 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T16:38:01.466 INFO:teuthology.orchestra.run.smithi003.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 236 kB/s | 32 kB 00:00 2024-03-25T16:38:01.499 INFO:teuthology.orchestra.run.smithi003.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 525 kB/s | 17 kB 00:00 2024-03-25T16:38:01.575 INFO:teuthology.orchestra.run.smithi003.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-25T16:38:01.600 INFO:teuthology.orchestra.run.smithi003.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 348 kB/s | 34 kB 00:00 2024-03-25T16:38:01.776 INFO:teuthology.orchestra.run.smithi003.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 24 MB/s | 4.7 MB 00:00 2024-03-25T16:38:01.834 INFO:teuthology.orchestra.run.smithi003.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-03-25T16:38:01.859 INFO:teuthology.orchestra.run.smithi003.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-03-25T16:38:02.027 INFO:teuthology.orchestra.run.smithi003.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.8 MB/s | 803 kB 00:00 2024-03-25T16:38:02.185 INFO:teuthology.orchestra.run.smithi003.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-03-25T16:38:02.210 INFO:teuthology.orchestra.run.smithi003.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-03-25T16:38:02.461 INFO:teuthology.orchestra.run.smithi003.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 893 kB/s | 245 kB 00:00 2024-03-25T16:38:12.250 INFO:teuthology.orchestra.run.smithi016.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 309 kB/s | 17 MB 00:55 2024-03-25T16:38:12.254 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:12.254 INFO:teuthology.orchestra.run.smithi016.stdout:Total 503 kB/s | 30 MB 01:00 2024-03-25T16:38:12.275 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T16:38:12.319 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T16:38:12.319 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T16:38:12.746 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T16:38:12.747 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T16:38:14.101 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T16:38:14.152 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:38:14.190 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:38:14.221 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:38:14.296 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:38:14.337 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:38:14.370 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:38:14.482 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:38:14.559 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:38:14.607 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:38:14.682 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:38:14.734 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:38:14.771 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:38:14.829 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:38:14.893 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:38:15.087 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:38:15.134 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:38:15.172 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:38:15.223 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:38:15.253 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:38:15.332 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:38:15.363 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:38:15.397 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:38:15.437 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:38:15.473 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:38:15.515 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:38:15.608 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:38:15.646 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:38:15.722 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:38:15.872 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:38:16.003 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:38:16.170 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:38:16.201 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:38:16.225 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:38:16.225 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:38:16.226 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:38:16.226 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:38:16.226 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:38:16.226 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:38:17.746 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:38:17.778 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:38:17.778 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:38:17.778 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:38:17.778 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:38:17.778 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:38:17.778 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:38:18.237 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:38:18.259 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:38:18.259 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:38:18.260 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:38:18.260 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:38:18.260 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:38:18.260 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:38:18.471 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:38:18.493 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:38:18.494 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:38:18.494 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:38:18.494 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:38:18.494 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:38:18.494 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:38:18.563 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:38:20.173 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:38:20.174 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:38:20.174 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:38:20.174 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:38:20.174 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:38:20.174 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:38:20.174 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:38:20.174 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:38:20.174 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:38:20.174 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:38:20.174 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:38:20.174 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:38:20.175 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:38:20.175 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:38:20.175 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:38:20.175 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:38:20.175 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:38:20.175 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:38:20.175 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:38:20.175 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:38:20.175 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:38:20.175 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:38:20.175 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:38:20.175 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:38:20.175 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:38:20.175 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:38:20.177 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:38:20.177 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:38:20.177 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:38:20.177 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:38:20.177 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:38:20.177 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:38:20.177 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:38:20.177 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:38:20.177 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:38:20.177 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:38:21.162 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:38:21.162 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:38:21.162 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T16:38:21.162 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:21.163 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:21.163 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:21.163 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:38:21.163 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:21.163 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:21.163 INFO:teuthology.orchestra.run.smithi016.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:38:21.163 INFO:teuthology.orchestra.run.smithi016.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:38:21.163 INFO:teuthology.orchestra.run.smithi016.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:38:21.163 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:38:21.163 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:38:21.163 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:38:21.163 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:38:21.163 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:38:21.164 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:38:21.164 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:38:21.164 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:38:21.164 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:38:21.164 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:38:21.164 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:38:21.164 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:38:21.164 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:38:21.164 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:38:21.164 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:38:21.164 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:38:21.164 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:38:21.164 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:38:21.164 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:38:21.164 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:38:21.165 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:38:21.165 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:38:21.165 INFO:teuthology.orchestra.run.smithi016.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:38:21.165 INFO:teuthology.orchestra.run.smithi016.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:38:21.165 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:38:21.165 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:38:21.165 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:38:21.165 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:38:21.165 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T16:38:21.533 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-base 2024-03-25T16:38:22.121 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:05:03 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T16:38:22.221 INFO:teuthology.orchestra.run.smithi016.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:38:22.278 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T16:38:22.279 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-03-25T16:38:22.279 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T16:38:22.342 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install cephadm 2024-03-25T16:38:22.915 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:05:03 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T16:38:23.070 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T16:38:23.070 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:38:23.071 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-03-25T16:38:23.071 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:38:23.071 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T16:38:23.071 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:38:23.071 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:38:23.071 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T16:38:23.071 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:38:23.071 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-25T16:38:23.071 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:38:23.071 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 747 k 2024-03-25T16:38:23.071 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 754 k 2024-03-25T16:38:23.071 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T16:38:26.624 INFO:teuthology.orchestra.run.smithi016.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 210 kB/s | 747 kB 00:03 2024-03-25T16:38:26.625 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:26.625 INFO:teuthology.orchestra.run.smithi016.stdout:Total 210 kB/s | 747 kB 00:03 2024-03-25T16:38:26.625 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T16:38:26.632 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T16:38:26.632 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T16:38:26.638 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T16:38:26.638 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T16:38:26.712 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T16:38:27.170 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:38:27.265 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:38:27.627 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:38:27.649 INFO:teuthology.orchestra.run.smithi003.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 288 kB/s | 7.4 MB 00:26 2024-03-25T16:38:27.652 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:27.652 INFO:teuthology.orchestra.run.smithi003.stdout:Total 1.4 MB/s | 37 MB 00:26 2024-03-25T16:38:27.695 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T16:38:27.723 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T16:38:27.788 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T16:38:27.899 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:38:27.899 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:38:27.899 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T16:38:27.899 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:38:27.899 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:38:27.899 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T16:38:28.107 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T16:38:28.108 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T16:38:28.146 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:38:28.726 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:05:09 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T16:38:28.838 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T16:38:28.883 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T16:38:28.884 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:38:28.884 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-03-25T16:38:28.884 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:38:28.884 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T16:38:28.885 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:38:28.885 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:38:28.885 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T16:38:28.885 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:38:28.885 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-25T16:38:28.885 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:38:28.885 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 145 k 2024-03-25T16:38:28.885 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 436 k 2024-03-25T16:38:28.885 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T16:38:28.906 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:38:28.937 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:38:29.036 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:38:29.082 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:38:29.266 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:38:29.289 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:38:29.373 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:38:29.715 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 175 kB/s | 145 kB 00:00 2024-03-25T16:38:29.715 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:29.715 INFO:teuthology.orchestra.run.smithi016.stdout:Total 175 kB/s | 145 kB 00:00 2024-03-25T16:38:29.716 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T16:38:29.725 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T16:38:29.725 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T16:38:29.784 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T16:38:29.784 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T16:38:29.932 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T16:38:29.999 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:38:30.023 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:38:30.023 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:38:30.023 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:38:30.023 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:38:30.067 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:38:30.189 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:38:31.019 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:38:31.019 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:38:31.019 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T16:38:31.019 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:31.019 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:38:31.019 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T16:38:31.255 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr 2024-03-25T16:38:31.485 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:38:31.832 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:05:12 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T16:38:31.932 INFO:teuthology.orchestra.run.smithi016.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:38:31.987 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T16:38:31.988 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-03-25T16:38:31.988 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T16:38:32.053 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:38:32.551 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:38:32.579 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:38:32.630 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:05:13 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T16:38:32.785 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T16:38:32.786 INFO:teuthology.orchestra.run.smithi016.stdout:====================================================================================== 2024-03-25T16:38:32.786 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-03-25T16:38:32.786 INFO:teuthology.orchestra.run.smithi016.stdout:====================================================================================== 2024-03-25T16:38:32.787 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T16:38:32.787 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:38:32.787 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-03-25T16:38:32.787 INFO:teuthology.orchestra.run.smithi016.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:38:32.787 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:38:32.787 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:38:32.787 INFO:teuthology.orchestra.run.smithi016.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:38:32.787 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:38:32.787 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:38:32.787 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:38:32.787 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:38:32.787 INFO:teuthology.orchestra.run.smithi016.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:38:32.787 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:38:32.788 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:38:32.788 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:38:32.788 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T16:38:32.788 INFO:teuthology.orchestra.run.smithi016.stdout:====================================================================================== 2024-03-25T16:38:32.788 INFO:teuthology.orchestra.run.smithi016.stdout:Install 12 Packages 2024-03-25T16:38:32.788 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:38:32.788 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 6.9 M 2024-03-25T16:38:32.789 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 26 M 2024-03-25T16:38:32.789 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T16:38:33.719 INFO:teuthology.orchestra.run.smithi016.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 60 kB/s | 31 kB 00:00 2024-03-25T16:38:33.778 INFO:teuthology.orchestra.run.smithi016.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 26 kB/s | 15 kB 00:00 2024-03-25T16:38:33.870 INFO:teuthology.orchestra.run.smithi016.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.6 MB/s | 238 kB 00:00 2024-03-25T16:38:33.903 INFO:teuthology.orchestra.run.smithi016.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.5 MB/s | 1.0 MB 00:00 2024-03-25T16:38:34.095 INFO:teuthology.orchestra.run.smithi016.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.8 MB/s | 863 kB 00:00 2024-03-25T16:38:34.353 INFO:teuthology.orchestra.run.smithi016.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 75 kB/s | 19 kB 00:00 2024-03-25T16:38:34.454 INFO:teuthology.orchestra.run.smithi016.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 995 kB/s | 548 kB 00:00 2024-03-25T16:38:34.555 INFO:teuthology.orchestra.run.smithi016.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.4 MB/s | 144 kB 00:00 2024-03-25T16:38:34.647 INFO:teuthology.orchestra.run.smithi016.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 336 kB/s | 31 kB 00:00 2024-03-25T16:38:34.739 INFO:teuthology.orchestra.run.smithi016.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-03-25T16:38:34.897 INFO:teuthology.orchestra.run.smithi016.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.7 MB/s | 2.0 MB 00:00 2024-03-25T16:38:35.029 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:38:35.029 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:38:35.029 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:38:35.030 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:38:35.030 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:38:35.030 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:38:35.030 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:38:35.030 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:38:35.030 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:38:35.030 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:38:35.030 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:38:35.030 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:38:35.525 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:38:35.525 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:38:35.525 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T16:38:35.526 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:38:35.526 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:38:35.526 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:38:35.526 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:38:35.526 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:38:35.526 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:38:35.526 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:38:35.526 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:38:35.526 INFO:teuthology.orchestra.run.smithi003.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:38:35.526 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:38:35.526 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:38:35.526 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:38:35.526 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:38:35.527 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T16:38:35.897 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:38:36.473 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:05:34 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T16:38:36.628 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T16:38:36.629 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2024-03-25T16:38:36.629 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-25T16:38:36.630 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2024-03-25T16:38:36.630 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T16:38:36.630 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:38:36.630 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-25T16:38:36.630 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:38:36.630 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:38:36.630 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:38:36.630 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:38:36.630 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:38:36.630 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:38:36.630 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:38:36.630 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:38:36.631 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:38:36.631 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:38:36.631 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:38:36.631 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:38:36.631 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:38:36.631 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T16:38:36.631 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2024-03-25T16:38:36.631 INFO:teuthology.orchestra.run.smithi003.stdout:Install 13 Packages 2024-03-25T16:38:36.631 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:38:36.632 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 2.0 M 2024-03-25T16:38:36.632 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 26 M 2024-03-25T16:38:36.632 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T16:38:36.887 INFO:teuthology.orchestra.run.smithi003.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 239 kB/s | 24 kB 00:00 2024-03-25T16:38:36.937 INFO:teuthology.orchestra.run.smithi003.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 119 kB/s | 18 kB 00:00 2024-03-25T16:38:36.987 INFO:teuthology.orchestra.run.smithi003.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 174 kB 00:00 2024-03-25T16:38:37.012 INFO:teuthology.orchestra.run.smithi003.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T16:38:37.038 INFO:teuthology.orchestra.run.smithi003.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T16:38:37.063 INFO:teuthology.orchestra.run.smithi003.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 872 kB/s | 43 kB 00:00 2024-03-25T16:38:37.130 INFO:teuthology.orchestra.run.smithi003.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 224 kB/s | 15 kB 00:00 2024-03-25T16:38:37.155 INFO:teuthology.orchestra.run.smithi003.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 275 kB/s | 32 kB 00:00 2024-03-25T16:38:37.230 INFO:teuthology.orchestra.run.smithi003.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 2.0 MB/s | 203 kB 00:00 2024-03-25T16:38:37.273 INFO:teuthology.orchestra.run.smithi003.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2024-03-25T16:38:37.314 INFO:teuthology.orchestra.run.smithi003.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.1 MB/s | 90 kB 00:00 2024-03-25T16:38:37.415 INFO:teuthology.orchestra.run.smithi003.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 3.9 MB/s | 1.0 MB 00:00 2024-03-25T16:38:37.440 INFO:teuthology.orchestra.run.smithi003.stdout:(13/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 80 kB/s | 52 kB 00:00 2024-03-25T16:38:37.441 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:37.441 INFO:teuthology.orchestra.run.smithi003.stdout:Total 2.5 MB/s | 2.0 MB 00:00 2024-03-25T16:38:37.442 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T16:38:37.456 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T16:38:37.456 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T16:38:37.553 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T16:38:37.555 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T16:38:38.100 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T16:38:38.277 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:38:38.334 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:38:38.398 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:38:38.434 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:38:38.475 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:38:38.572 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:38:38.681 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:38:38.735 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:38:39.492 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:38:39.542 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:38:39.576 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:38:39.625 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:38:39.653 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:38:40.488 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:38:40.488 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:38:40.488 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:38:40.488 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:38:40.488 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:38:40.489 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:38:40.489 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:38:40.489 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:38:40.489 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:38:40.489 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:38:40.489 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:38:40.490 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:38:40.490 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:38:40.847 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:38:40.847 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:38:40.847 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T16:38:40.847 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:38:40.847 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:38:40.847 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:38:40.847 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:38:40.847 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:38:40.847 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:38:40.847 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:38:40.847 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:38:40.847 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:38:40.848 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:38:40.848 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:38:40.848 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:38:40.848 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:38:40.848 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:38:40.848 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T16:38:41.105 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:38:41.173 INFO:teuthology.orchestra.run.smithi016.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 229 kB/s | 1.8 MB 00:07 2024-03-25T16:38:41.175 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:41.175 INFO:teuthology.orchestra.run.smithi016.stdout:Total 837 kB/s | 6.9 MB 00:08 2024-03-25T16:38:41.196 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T16:38:41.217 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T16:38:41.217 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T16:38:41.395 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T16:38:41.395 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T16:38:41.680 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:05:39 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T16:38:41.836 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T16:38:41.837 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2024-03-25T16:38:41.837 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-25T16:38:41.837 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2024-03-25T16:38:41.837 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T16:38:41.838 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:38:41.838 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-25T16:38:41.838 INFO:teuthology.orchestra.run.smithi003.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:38:41.838 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:38:41.838 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:38:41.838 INFO:teuthology.orchestra.run.smithi003.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:38:41.838 INFO:teuthology.orchestra.run.smithi003.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:38:41.838 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:38:41.838 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T16:38:41.838 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2024-03-25T16:38:41.838 INFO:teuthology.orchestra.run.smithi003.stdout:Install 6 Packages 2024-03-25T16:38:41.838 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:38:41.839 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 6.8 M 2024-03-25T16:38:41.839 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 33 M 2024-03-25T16:38:41.839 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T16:38:42.033 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T16:38:42.115 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:38:42.150 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:38:42.197 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:38:42.226 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:38:42.243 INFO:teuthology.orchestra.run.smithi003.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-03-25T16:38:42.318 INFO:teuthology.orchestra.run.smithi003.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-25T16:38:42.339 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:38:42.474 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:38:42.493 INFO:teuthology.orchestra.run.smithi003.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 332 kB/s | 58 kB 00:00 2024-03-25T16:38:42.560 INFO:teuthology.orchestra.run.smithi003.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.7 MB/s | 548 kB 00:00 2024-03-25T16:38:42.585 INFO:teuthology.orchestra.run.smithi003.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 557 kB/s | 51 kB 00:00 2024-03-25T16:38:42.698 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:38:42.749 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:38:42.777 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:38:42.800 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:38:43.049 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:38:43.083 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:38:43.277 INFO:teuthology.orchestra.run.smithi003.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 111 kB/s | 138 kB 00:01 2024-03-25T16:38:43.281 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:43.281 INFO:teuthology.orchestra.run.smithi003.stdout:Total 4.7 MB/s | 6.8 MB 00:01 2024-03-25T16:38:43.295 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T16:38:43.310 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T16:38:43.310 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T16:38:43.403 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T16:38:43.404 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T16:38:43.702 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T16:38:43.817 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:38:43.817 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:38:43.817 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:38:43.817 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:38:43.817 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:38:43.817 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:38:43.817 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:38:43.817 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:38:43.817 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:38:43.817 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:38:43.818 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:38:43.818 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:38:43.876 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:38:43.921 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:38:44.154 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:38:44.154 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:38:44.154 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T16:38:44.155 INFO:teuthology.orchestra.run.smithi016.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:38:44.155 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:38:44.155 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:38:44.155 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:38:44.155 INFO:teuthology.orchestra.run.smithi016.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:38:44.155 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:38:44.155 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:38:44.155 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:38:44.155 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:38:44.155 INFO:teuthology.orchestra.run.smithi016.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:38:44.155 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:38:44.156 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:38:44.156 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:38:44.156 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T16:38:44.419 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:38:44.476 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:38:44.560 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:38:44.628 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:38:44.657 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:38:44.989 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:05:25 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T16:38:45.146 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T16:38:45.149 INFO:teuthology.orchestra.run.smithi016.stdout:============================================================================================ 2024-03-25T16:38:45.150 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-03-25T16:38:45.150 INFO:teuthology.orchestra.run.smithi016.stdout:============================================================================================ 2024-03-25T16:38:45.150 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T16:38:45.150 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:38:45.150 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-03-25T16:38:45.150 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:38:45.150 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:38:45.150 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:38:45.150 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:38:45.150 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:38:45.150 INFO:teuthology.orchestra.run.smithi016.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:38:45.151 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:38:45.151 INFO:teuthology.orchestra.run.smithi016.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:38:45.151 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:38:45.151 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:38:45.151 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:38:45.151 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:38:45.151 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T16:38:45.151 INFO:teuthology.orchestra.run.smithi016.stdout:============================================================================================ 2024-03-25T16:38:45.151 INFO:teuthology.orchestra.run.smithi016.stdout:Install 12 Packages 2024-03-25T16:38:45.151 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:38:45.151 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 37 M 2024-03-25T16:38:45.151 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 209 M 2024-03-25T16:38:45.151 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T16:38:45.452 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:38:45.452 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:38:45.452 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:38:45.452 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:38:45.452 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:38:45.452 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:38:45.479 INFO:teuthology.orchestra.run.smithi016.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 275 kB/s | 32 kB 00:00 2024-03-25T16:38:45.529 INFO:teuthology.orchestra.run.smithi016.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 670 kB/s | 17 kB 00:00 2024-03-25T16:38:45.554 INFO:teuthology.orchestra.run.smithi016.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.3 MB/s | 34 kB 00:00 2024-03-25T16:38:45.764 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:38:45.764 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:38:45.764 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T16:38:45.764 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:38:45.764 INFO:teuthology.orchestra.run.smithi003.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:38:45.764 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:38:45.764 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:38:45.765 INFO:teuthology.orchestra.run.smithi003.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:38:45.765 INFO:teuthology.orchestra.run.smithi003.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:38:45.765 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:38:45.765 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T16:38:45.771 INFO:teuthology.orchestra.run.smithi016.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 22 MB/s | 4.7 MB 00:00 2024-03-25T16:38:45.963 INFO:teuthology.orchestra.run.smithi016.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 4.8 MB 00:00 2024-03-25T16:38:45.993 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-fuse 2024-03-25T16:38:46.014 INFO:teuthology.orchestra.run.smithi016.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.6 MB/s | 3.0 MB 00:00 2024-03-25T16:38:46.039 INFO:teuthology.orchestra.run.smithi016.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 217 kB 00:00 2024-03-25T16:38:46.514 INFO:teuthology.orchestra.run.smithi016.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-03-25T16:38:46.556 INFO:teuthology.orchestra.run.smithi016.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2024-03-25T16:38:46.567 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:05:44 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T16:38:46.706 INFO:teuthology.orchestra.run.smithi016.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2024-03-25T16:38:46.724 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T16:38:46.724 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:38:46.724 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-25T16:38:46.725 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:38:46.725 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T16:38:46.725 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:38:46.725 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-25T16:38:46.725 INFO:teuthology.orchestra.run.smithi003.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:38:46.725 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:38:46.725 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T16:38:46.725 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:38:46.725 INFO:teuthology.orchestra.run.smithi003.stdout:Install 2 Packages 2024-03-25T16:38:46.725 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:38:46.726 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 964 k 2024-03-25T16:38:46.726 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 2.8 M 2024-03-25T16:38:46.726 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T16:38:46.731 INFO:teuthology.orchestra.run.smithi016.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 901 kB/s | 195 kB 00:00 2024-03-25T16:38:46.886 INFO:teuthology.orchestra.run.smithi003.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 490 kB/s | 79 kB 00:00 2024-03-25T16:38:50.478 INFO:teuthology.orchestra.run.smithi003.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 236 kB/s | 885 kB 00:03 2024-03-25T16:38:50.479 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:50.480 INFO:teuthology.orchestra.run.smithi003.stdout:Total 257 kB/s | 964 kB 00:03 2024-03-25T16:38:50.480 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T16:38:50.504 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T16:38:50.504 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T16:38:50.584 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T16:38:50.584 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T16:38:50.772 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T16:38:50.925 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:38:51.005 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:38:51.659 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:38:51.659 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:38:51.933 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:38:51.934 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:38:51.934 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T16:38:51.934 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:38:51.934 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:38:51.934 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T16:38:52.239 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-volume 2024-03-25T16:38:52.816 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:05:50 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T16:38:52.972 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T16:38:52.973 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:38:52.973 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-25T16:38:52.973 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:38:52.973 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T16:38:52.973 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:38:52.973 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-25T16:38:52.973 INFO:teuthology.orchestra.run.smithi003.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:38:52.974 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:38:52.974 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T16:38:52.974 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:38:52.974 INFO:teuthology.orchestra.run.smithi003.stdout:Install 2 Packages 2024-03-25T16:38:52.974 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:38:52.974 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 332 k 2024-03-25T16:38:52.974 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 1.5 M 2024-03-25T16:38:52.974 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T16:38:53.082 INFO:teuthology.orchestra.run.smithi003.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 639 kB/s | 69 kB 00:00 2024-03-25T16:38:54.224 INFO:teuthology.orchestra.run.smithi003.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 211 kB/s | 264 kB 00:01 2024-03-25T16:38:54.225 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:54.226 INFO:teuthology.orchestra.run.smithi003.stdout:Total 265 kB/s | 332 kB 00:01 2024-03-25T16:38:54.226 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T16:38:54.237 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T16:38:54.237 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T16:38:54.285 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T16:38:54.285 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T16:38:54.444 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T16:38:54.629 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:38:54.668 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:38:54.683 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:38:54.683 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:38:54.683 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:38:54.683 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:38:55.438 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:38:55.778 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:38:55.778 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:38:55.778 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T16:38:55.778 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:38:55.778 INFO:teuthology.orchestra.run.smithi003.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:38:55.779 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:38:55.779 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T16:38:56.105 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librados-devel 2024-03-25T16:38:56.686 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:05:54 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T16:38:56.841 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T16:38:56.842 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:38:56.842 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-25T16:38:56.842 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:38:56.842 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T16:38:56.842 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:38:56.842 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:38:56.842 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T16:38:56.842 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:38:56.842 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-25T16:38:56.842 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:38:56.843 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 127 k 2024-03-25T16:38:56.843 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 456 k 2024-03-25T16:38:56.843 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T16:38:57.762 INFO:teuthology.orchestra.run.smithi003.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 138 kB/s | 127 kB 00:00 2024-03-25T16:38:57.762 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:57.763 INFO:teuthology.orchestra.run.smithi003.stdout:Total 137 kB/s | 127 kB 00:00 2024-03-25T16:38:57.763 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T16:38:57.770 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T16:38:57.770 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T16:38:57.833 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T16:38:57.833 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T16:38:58.117 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T16:38:58.230 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:58.804 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:59.126 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:59.127 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:38:59.127 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T16:38:59.127 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:59.127 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:38:59.127 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T16:38:59.346 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install libcephfs2 2024-03-25T16:38:59.922 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:05:57 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T16:39:00.022 INFO:teuthology.orchestra.run.smithi003.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:39:00.079 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T16:39:00.080 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-25T16:39:00.080 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T16:39:00.152 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install libcephfs-devel 2024-03-25T16:39:00.730 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:05:58 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T16:39:00.885 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T16:39:00.886 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:39:00.886 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-25T16:39:00.886 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:39:00.886 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T16:39:00.886 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:39:00.886 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:39:00.886 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T16:39:00.887 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:39:00.887 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-25T16:39:00.887 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:39:00.887 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 31 k 2024-03-25T16:39:00.887 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 140 k 2024-03-25T16:39:00.887 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T16:39:01.340 INFO:teuthology.orchestra.run.smithi003.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 69 kB/s | 31 kB 00:00 2024-03-25T16:39:01.340 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:01.341 INFO:teuthology.orchestra.run.smithi003.stdout:Total 69 kB/s | 31 kB 00:00 2024-03-25T16:39:01.341 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T16:39:01.344 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T16:39:01.344 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T16:39:01.373 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T16:39:01.374 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T16:39:01.496 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T16:39:01.608 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:01.963 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:02.223 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:02.223 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:39:02.223 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T16:39:02.223 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:39:02.223 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:39:02.223 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T16:39:02.392 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librados2 2024-03-25T16:39:02.969 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:06:00 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T16:39:03.070 INFO:teuthology.orchestra.run.smithi003.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:39:03.126 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T16:39:03.127 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-25T16:39:03.127 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T16:39:03.189 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librbd1 2024-03-25T16:39:03.779 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:06:01 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T16:39:03.880 INFO:teuthology.orchestra.run.smithi003.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:39:03.935 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T16:39:03.936 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-25T16:39:03.936 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T16:39:03.990 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rados 2024-03-25T16:39:04.573 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:06:02 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T16:39:04.672 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:39:04.726 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T16:39:04.727 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-25T16:39:04.727 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T16:39:04.778 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rgw 2024-03-25T16:39:05.351 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:06:03 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T16:39:05.450 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:39:05.503 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T16:39:05.505 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-25T16:39:05.505 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T16:39:05.562 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-cephfs 2024-03-25T16:39:06.140 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:06:04 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T16:39:06.240 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:39:06.295 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T16:39:06.296 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-25T16:39:06.296 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T16:39:06.344 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rbd 2024-03-25T16:39:06.918 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:06:04 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T16:39:07.018 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:39:07.072 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T16:39:07.073 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-25T16:39:07.073 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T16:39:07.130 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-fuse 2024-03-25T16:39:07.705 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:06:05 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T16:39:07.860 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T16:39:07.861 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:39:07.861 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2024-03-25T16:39:07.861 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:39:07.861 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T16:39:07.861 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:39:07.861 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:39:07.861 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T16:39:07.861 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:39:07.861 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-25T16:39:07.861 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:39:07.861 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 85 k 2024-03-25T16:39:07.861 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 231 k 2024-03-25T16:39:07.862 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T16:39:08.860 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 85 kB/s | 85 kB 00:00 2024-03-25T16:39:08.860 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:08.860 INFO:teuthology.orchestra.run.smithi003.stdout:Total 85 kB/s | 85 kB 00:00 2024-03-25T16:39:08.861 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T16:39:08.869 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T16:39:08.869 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T16:39:08.931 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T16:39:08.931 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T16:39:09.075 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T16:39:09.185 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:09.573 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:09.897 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:09.897 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:39:09.897 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T16:39:09.897 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:39:09.897 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:39:09.898 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T16:39:10.147 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-mirror 2024-03-25T16:39:10.724 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:06:08 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T16:39:10.880 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T16:39:10.880 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:39:10.880 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-25T16:39:10.880 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:39:10.880 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T16:39:10.880 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:39:10.880 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:39:10.881 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T16:39:10.881 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:39:10.881 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-25T16:39:10.881 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:39:10.881 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 3.1 M 2024-03-25T16:39:10.881 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 13 M 2024-03-25T16:39:10.881 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T16:39:13.379 INFO:teuthology.orchestra.run.smithi016.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 270 kB/s | 7.4 MB 00:28 2024-03-25T16:39:13.381 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:13.381 INFO:teuthology.orchestra.run.smithi016.stdout:Total 1.3 MB/s | 37 MB 00:28 2024-03-25T16:39:13.424 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T16:39:13.451 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T16:39:13.451 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T16:39:13.831 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T16:39:13.831 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T16:39:14.551 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T16:39:14.661 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:39:14.692 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:39:14.790 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:39:14.837 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:39:15.021 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:39:15.052 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:39:15.135 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:39:15.811 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:39:15.919 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:39:17.228 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:39:18.420 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:39:18.444 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:39:18.903 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 394 kB/s | 3.1 MB 00:08 2024-03-25T16:39:18.904 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:18.904 INFO:teuthology.orchestra.run.smithi003.stdout:Total 394 kB/s | 3.1 MB 00:08 2024-03-25T16:39:18.904 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T16:39:18.916 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T16:39:18.916 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T16:39:18.996 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T16:39:18.996 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T16:39:19.447 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T16:39:19.516 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:19.541 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:19.541 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:39:19.541 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:39:19.541 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:39:19.541 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:39:19.541 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:39:20.632 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:20.632 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:39:20.632 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T16:39:20.632 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:39:20.632 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:39:20.632 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T16:39:20.881 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-nbd 2024-03-25T16:39:20.904 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:39:20.904 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:39:20.904 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:39:20.904 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:39:20.904 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:39:20.904 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:39:20.904 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:39:20.905 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:39:20.905 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:39:20.905 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:39:20.905 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:39:20.905 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:39:21.404 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:39:21.404 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:21.404 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T16:39:21.404 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:39:21.404 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:39:21.404 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:39:21.404 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:39:21.404 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:39:21.404 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:39:21.404 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:39:21.404 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:39:21.404 INFO:teuthology.orchestra.run.smithi016.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:39:21.404 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:39:21.405 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:39:21.405 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:39:21.405 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:21.405 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T16:39:21.460 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:06:19 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T16:39:21.615 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T16:39:21.615 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:39:21.616 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2024-03-25T16:39:21.616 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:39:21.616 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T16:39:21.616 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:39:21.616 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:39:21.616 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T16:39:21.616 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:39:21.616 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-25T16:39:21.616 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:39:21.616 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 171 k 2024-03-25T16:39:21.616 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 494 k 2024-03-25T16:39:21.616 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T16:39:21.834 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:39:22.410 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:06:03 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T16:39:22.564 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 181 kB/s | 171 kB 00:00 2024-03-25T16:39:22.565 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:22.565 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T16:39:22.566 INFO:teuthology.orchestra.run.smithi003.stdout:Total 181 kB/s | 171 kB 00:00 2024-03-25T16:39:22.566 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T16:39:22.567 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================== 2024-03-25T16:39:22.567 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-03-25T16:39:22.567 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================== 2024-03-25T16:39:22.567 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T16:39:22.567 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:39:22.567 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-03-25T16:39:22.567 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:39:22.567 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:39:22.567 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:39:22.567 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:39:22.567 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:39:22.568 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:39:22.568 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:39:22.568 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:39:22.568 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:39:22.568 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:39:22.568 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:39:22.568 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:39:22.568 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:22.568 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T16:39:22.568 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================== 2024-03-25T16:39:22.568 INFO:teuthology.orchestra.run.smithi016.stdout:Install 13 Packages 2024-03-25T16:39:22.568 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:22.569 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 2.0 M 2024-03-25T16:39:22.569 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 26 M 2024-03-25T16:39:22.569 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T16:39:22.575 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T16:39:22.575 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T16:39:22.636 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T16:39:22.636 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T16:39:22.720 INFO:teuthology.orchestra.run.smithi016.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 233 kB/s | 24 kB 00:00 2024-03-25T16:39:22.770 INFO:teuthology.orchestra.run.smithi016.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 117 kB/s | 18 kB 00:00 2024-03-25T16:39:22.792 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T16:39:22.821 INFO:teuthology.orchestra.run.smithi016.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 174 kB 00:00 2024-03-25T16:39:22.846 INFO:teuthology.orchestra.run.smithi016.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T16:39:22.872 INFO:teuthology.orchestra.run.smithi016.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T16:39:22.897 INFO:teuthology.orchestra.run.smithi016.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 872 kB/s | 43 kB 00:00 2024-03-25T16:39:22.908 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:23.031 INFO:teuthology.orchestra.run.smithi016.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 203 kB/s | 32 kB 00:00 2024-03-25T16:39:23.056 INFO:teuthology.orchestra.run.smithi016.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 94 kB/s | 15 kB 00:00 2024-03-25T16:39:23.198 INFO:teuthology.orchestra.run.smithi016.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 1.2 MB/s | 203 kB 00:00 2024-03-25T16:39:23.298 INFO:teuthology.orchestra.run.smithi016.stdout:(10/13): python3-kubernetes-26.1.0-2.el9.noarch 4.7 MB/s | 1.0 MB 00:00 2024-03-25T16:39:23.324 INFO:teuthology.orchestra.run.smithi016.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 473 kB/s | 59 kB 00:00 2024-03-25T16:39:23.351 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:23.374 INFO:teuthology.orchestra.run.smithi016.stdout:(12/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 69 kB/s | 52 kB 00:00 2024-03-25T16:39:23.441 INFO:teuthology.orchestra.run.smithi016.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 632 kB/s | 90 kB 00:00 2024-03-25T16:39:23.443 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:23.443 INFO:teuthology.orchestra.run.smithi016.stdout:Total 2.3 MB/s | 2.0 MB 00:00 2024-03-25T16:39:23.443 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T16:39:23.457 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T16:39:23.457 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T16:39:23.552 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T16:39:23.553 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T16:39:23.640 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:23.640 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:39:23.640 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T16:39:23.640 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:39:23.640 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:39:23.640 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T16:39:23.882 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:39:24.071 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T16:39:24.236 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:39:24.294 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:39:24.349 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:39:24.385 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:39:24.427 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:39:24.459 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:06:22 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T16:39:24.525 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:39:24.615 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T16:39:24.616 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:39:24.616 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-25T16:39:24.616 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:39:24.616 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T16:39:24.616 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:39:24.616 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-25T16:39:24.616 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:39:24.616 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:39:24.616 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T16:39:24.616 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:39:24.616 INFO:teuthology.orchestra.run.smithi003.stdout:Install 2 Packages 2024-03-25T16:39:24.616 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:39:24.617 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 91 M 2024-03-25T16:39:24.617 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 498 M 2024-03-25T16:39:24.617 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T16:39:24.633 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:39:24.695 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:39:25.452 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:39:25.508 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:39:25.542 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:39:25.591 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:39:25.620 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:39:26.405 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:39:26.406 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:39:26.406 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:39:26.406 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:39:26.406 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:39:26.406 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:39:26.406 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:39:26.406 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:39:26.406 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:39:26.406 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:39:26.406 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:39:26.406 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:39:26.406 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:39:26.766 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:39:26.790 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:26.790 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T16:39:26.790 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:39:26.790 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:39:26.790 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:39:26.790 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:39:26.790 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:39:26.790 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:39:26.790 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:39:26.790 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:39:26.790 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:39:26.790 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:39:26.791 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:39:26.791 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:39:26.791 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:39:26.791 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:26.791 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T16:39:27.017 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:39:27.594 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:06:08 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T16:39:27.751 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T16:39:27.752 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================== 2024-03-25T16:39:27.752 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-03-25T16:39:27.752 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================== 2024-03-25T16:39:27.752 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T16:39:27.752 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:39:27.752 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-03-25T16:39:27.752 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:39:27.752 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:39:27.752 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:39:27.753 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:39:27.753 INFO:teuthology.orchestra.run.smithi016.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:39:27.753 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:27.753 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T16:39:27.753 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================== 2024-03-25T16:39:27.753 INFO:teuthology.orchestra.run.smithi016.stdout:Install 6 Packages 2024-03-25T16:39:27.753 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:27.754 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 6.8 M 2024-03-25T16:39:27.754 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 33 M 2024-03-25T16:39:27.754 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T16:39:28.435 INFO:teuthology.orchestra.run.smithi016.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-03-25T16:39:28.519 INFO:teuthology.orchestra.run.smithi016.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-25T16:39:28.869 INFO:teuthology.orchestra.run.smithi016.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 165 kB/s | 58 kB 00:00 2024-03-25T16:39:28.928 INFO:teuthology.orchestra.run.smithi016.stdout:(4/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 195 kB/s | 138 kB 00:00 2024-03-25T16:39:29.011 INFO:teuthology.orchestra.run.smithi016.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 359 kB/s | 51 kB 00:00 2024-03-25T16:39:29.136 INFO:teuthology.orchestra.run.smithi016.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 786 kB/s | 548 kB 00:00 2024-03-25T16:39:29.138 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:29.139 INFO:teuthology.orchestra.run.smithi016.stdout:Total 4.9 MB/s | 6.8 MB 00:01 2024-03-25T16:39:29.153 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T16:39:29.166 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T16:39:29.166 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T16:39:29.260 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T16:39:29.261 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T16:39:29.566 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T16:39:29.742 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:39:29.789 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:39:30.359 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:39:30.451 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:39:30.512 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:39:30.542 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:39:31.330 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:39:31.330 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:39:31.330 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:39:31.330 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:39:31.330 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:39:31.330 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:39:31.624 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:39:31.624 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:31.624 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T16:39:31.624 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:39:31.624 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:39:31.624 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:39:31.624 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:39:31.624 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:39:31.624 INFO:teuthology.orchestra.run.smithi016.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:39:31.625 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:31.625 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T16:39:31.868 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-fuse 2024-03-25T16:39:32.447 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:06:13 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T16:39:32.602 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T16:39:32.604 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:39:32.604 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-03-25T16:39:32.604 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:39:32.604 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T16:39:32.604 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:39:32.604 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-03-25T16:39:32.604 INFO:teuthology.orchestra.run.smithi016.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:39:32.604 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:32.604 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T16:39:32.604 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:39:32.605 INFO:teuthology.orchestra.run.smithi016.stdout:Install 2 Packages 2024-03-25T16:39:32.605 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:32.605 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 964 k 2024-03-25T16:39:32.605 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 2.8 M 2024-03-25T16:39:32.605 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T16:39:32.730 INFO:teuthology.orchestra.run.smithi016.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 625 kB/s | 79 kB 00:00 2024-03-25T16:39:34.347 INFO:teuthology.orchestra.run.smithi016.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 508 kB/s | 885 kB 00:01 2024-03-25T16:39:34.349 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:34.349 INFO:teuthology.orchestra.run.smithi016.stdout:Total 552 kB/s | 964 kB 00:01 2024-03-25T16:39:34.350 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T16:39:34.375 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T16:39:34.375 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T16:39:34.457 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T16:39:34.457 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T16:39:34.648 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T16:39:34.786 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:39:34.867 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:39:35.513 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:39:35.513 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:39:35.797 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:39:35.797 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:35.797 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T16:39:35.797 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:39:35.797 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:35.798 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T16:39:36.076 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-volume 2024-03-25T16:39:36.657 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:06:17 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T16:39:36.812 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T16:39:36.812 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:39:36.813 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-03-25T16:39:36.813 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:39:36.813 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T16:39:36.813 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:39:36.813 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-03-25T16:39:36.813 INFO:teuthology.orchestra.run.smithi016.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:39:36.813 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:36.813 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T16:39:36.813 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:39:36.813 INFO:teuthology.orchestra.run.smithi016.stdout:Install 2 Packages 2024-03-25T16:39:36.813 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:36.814 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 332 k 2024-03-25T16:39:36.814 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 1.5 M 2024-03-25T16:39:36.814 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T16:39:36.936 INFO:teuthology.orchestra.run.smithi016.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 566 kB/s | 69 kB 00:00 2024-03-25T16:39:37.886 INFO:teuthology.orchestra.run.smithi016.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 246 kB/s | 264 kB 00:01 2024-03-25T16:39:37.887 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:37.888 INFO:teuthology.orchestra.run.smithi016.stdout:Total 309 kB/s | 332 kB 00:01 2024-03-25T16:39:37.888 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T16:39:37.899 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T16:39:37.899 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T16:39:37.941 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T16:39:37.942 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T16:39:38.101 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T16:39:38.283 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:39:38.316 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:39:38.331 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:39:38.331 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:39:38.331 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:39:38.331 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:39.090 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:39:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:39:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:39.423 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T16:39:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:39:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:39:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:39.424 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T16:39:39.719 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install librados-devel 2024-03-25T16:39:40.301 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:06:21 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T16:39:40.455 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T16:39:40.456 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:39:40.456 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-03-25T16:39:40.456 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:39:40.456 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T16:39:40.456 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:39:40.456 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:40.457 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T16:39:40.457 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:39:40.457 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-25T16:39:40.457 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:40.457 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 127 k 2024-03-25T16:39:40.457 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 456 k 2024-03-25T16:39:40.457 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T16:39:41.161 INFO:teuthology.orchestra.run.smithi016.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 180 kB/s | 127 kB 00:00 2024-03-25T16:39:41.161 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:41.162 INFO:teuthology.orchestra.run.smithi016.stdout:Total 180 kB/s | 127 kB 00:00 2024-03-25T16:39:41.162 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T16:39:41.168 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T16:39:41.168 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T16:39:41.228 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T16:39:41.229 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T16:39:41.387 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T16:39:41.496 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:41.915 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:42.200 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:42.200 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:42.200 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T16:39:42.200 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:39:42.200 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:42.200 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T16:39:42.436 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install libcephfs2 2024-03-25T16:39:43.017 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:06:24 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T16:39:43.117 INFO:teuthology.orchestra.run.smithi016.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:39:43.174 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T16:39:43.175 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-03-25T16:39:43.175 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T16:39:43.242 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install libcephfs-devel 2024-03-25T16:39:43.822 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:06:24 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T16:39:43.977 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T16:39:43.977 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:39:43.978 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-03-25T16:39:43.978 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:39:43.978 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T16:39:43.978 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:39:43.978 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:43.978 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T16:39:43.978 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:39:43.978 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-25T16:39:43.978 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:43.978 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 31 k 2024-03-25T16:39:43.978 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 140 k 2024-03-25T16:39:43.978 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T16:39:44.440 INFO:teuthology.orchestra.run.smithi016.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 68 kB/s | 31 kB 00:00 2024-03-25T16:39:44.440 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:44.441 INFO:teuthology.orchestra.run.smithi016.stdout:Total 67 kB/s | 31 kB 00:00 2024-03-25T16:39:44.441 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T16:39:44.445 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T16:39:44.445 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T16:39:44.471 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T16:39:44.472 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T16:39:44.592 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T16:39:44.716 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:45.064 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:45.322 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:45.322 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:45.322 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T16:39:45.322 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:39:45.322 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:45.322 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T16:39:45.472 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install librados2 2024-03-25T16:39:46.052 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:06:27 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T16:39:46.152 INFO:teuthology.orchestra.run.smithi016.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:39:46.207 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T16:39:46.208 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-03-25T16:39:46.208 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T16:39:46.262 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install librbd1 2024-03-25T16:39:46.837 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:06:27 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T16:39:46.937 INFO:teuthology.orchestra.run.smithi016.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:39:46.992 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T16:39:46.993 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-03-25T16:39:46.993 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T16:39:47.047 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-rados 2024-03-25T16:39:47.619 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:06:28 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T16:39:47.722 INFO:teuthology.orchestra.run.smithi016.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:39:47.778 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T16:39:47.778 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-03-25T16:39:47.778 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T16:39:47.828 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-rgw 2024-03-25T16:39:48.409 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:06:29 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T16:39:48.510 INFO:teuthology.orchestra.run.smithi016.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:39:48.565 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T16:39:48.567 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-03-25T16:39:48.567 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T16:39:48.623 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-cephfs 2024-03-25T16:39:49.208 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:06:30 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T16:39:49.308 INFO:teuthology.orchestra.run.smithi016.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:39:49.363 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T16:39:49.364 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-03-25T16:39:49.364 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T16:39:49.420 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-rbd 2024-03-25T16:39:49.997 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:06:30 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T16:39:50.096 INFO:teuthology.orchestra.run.smithi016.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:39:50.151 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T16:39:50.152 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-03-25T16:39:50.152 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T16:39:50.203 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install rbd-fuse 2024-03-25T16:39:50.780 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:06:31 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T16:39:50.934 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T16:39:50.935 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:39:50.935 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-03-25T16:39:50.935 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:39:50.935 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T16:39:50.935 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:39:50.935 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:50.935 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T16:39:50.935 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:39:50.935 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-25T16:39:50.935 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:50.936 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 85 k 2024-03-25T16:39:50.936 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 231 k 2024-03-25T16:39:50.936 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T16:39:51.560 INFO:teuthology.orchestra.run.smithi016.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 137 kB/s | 85 kB 00:00 2024-03-25T16:39:51.561 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:51.561 INFO:teuthology.orchestra.run.smithi016.stdout:Total 136 kB/s | 85 kB 00:00 2024-03-25T16:39:51.561 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T16:39:51.569 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T16:39:51.569 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T16:39:51.630 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T16:39:51.630 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T16:39:51.770 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T16:39:51.861 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:52.266 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:52.554 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:52.555 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:52.555 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T16:39:52.555 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:39:52.555 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:52.555 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T16:39:52.778 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install rbd-mirror 2024-03-25T16:39:53.357 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:06:34 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T16:39:53.511 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T16:39:53.511 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:39:53.511 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-03-25T16:39:53.511 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:39:53.511 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T16:39:53.511 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:39:53.512 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:53.512 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T16:39:53.512 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:39:53.512 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-25T16:39:53.512 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:39:53.512 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 3.1 M 2024-03-25T16:39:53.512 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 13 M 2024-03-25T16:39:53.512 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T16:40:07.738 INFO:teuthology.orchestra.run.smithi016.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 222 kB/s | 3.1 MB 00:14 2024-03-25T16:40:07.738 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:40:07.739 INFO:teuthology.orchestra.run.smithi016.stdout:Total 222 kB/s | 3.1 MB 00:14 2024-03-25T16:40:07.739 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T16:40:07.751 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T16:40:07.751 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T16:40:07.830 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T16:40:07.831 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T16:40:08.278 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T16:40:08.347 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:08.371 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:08.371 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:40:08.371 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:40:08.371 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:40:08.371 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:40:08.371 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:40:09.492 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:09.492 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:40:09.492 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T16:40:09.493 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:40:09.493 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:40:09.493 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T16:40:09.733 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install rbd-nbd 2024-03-25T16:40:10.315 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:06:51 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T16:40:10.470 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T16:40:10.471 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:40:10.471 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-03-25T16:40:10.471 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:40:10.472 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T16:40:10.472 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:40:10.472 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:40:10.472 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T16:40:10.472 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:40:10.472 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-25T16:40:10.472 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:40:10.472 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 171 k 2024-03-25T16:40:10.472 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 494 k 2024-03-25T16:40:10.472 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T16:40:11.616 INFO:teuthology.orchestra.run.smithi016.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 150 kB/s | 171 kB 00:01 2024-03-25T16:40:11.616 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:40:11.617 INFO:teuthology.orchestra.run.smithi016.stdout:Total 150 kB/s | 171 kB 00:01 2024-03-25T16:40:11.617 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T16:40:11.627 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T16:40:11.627 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T16:40:11.689 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T16:40:11.690 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T16:40:11.848 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T16:40:11.933 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:12.384 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:12.692 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:12.692 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:40:12.692 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T16:40:12.693 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:40:12.693 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:40:12.693 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T16:40:12.936 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:40:13.512 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:06:54 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T16:40:13.666 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T16:40:13.667 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:40:13.667 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-03-25T16:40:13.667 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:40:13.667 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T16:40:13.667 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:40:13.667 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-03-25T16:40:13.667 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:40:13.667 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:40:13.668 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T16:40:13.668 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:40:13.668 INFO:teuthology.orchestra.run.smithi016.stdout:Install 2 Packages 2024-03-25T16:40:13.668 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:40:13.668 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 91 M 2024-03-25T16:40:13.668 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 498 M 2024-03-25T16:40:13.669 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T16:40:43.057 INFO:teuthology.orchestra.run.smithi003.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 293 kB/s | 22 MB 01:18 2024-03-25T16:41:31.037 INFO:teuthology.orchestra.run.smithi016.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 297 kB/s | 22 MB 01:17 2024-03-25T16:42:20.744 INFO:teuthology.orchestra.run.smithi003.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 396 kB/s | 68 MB 02:56 2024-03-25T16:42:20.745 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:42:20.745 INFO:teuthology.orchestra.run.smithi003.stdout:Total 527 kB/s | 91 MB 02:56 2024-03-25T16:42:20.745 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T16:42:20.751 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T16:42:20.751 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T16:42:21.360 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T16:42:21.360 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T16:42:24.635 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T16:42:32.141 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:42:32.214 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:42:35.628 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:42:35.628 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:42:35.900 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:42:35.900 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:42:35.900 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T16:42:35.900 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:42:35.900 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:42:35.900 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:42:35.900 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T16:42:36.111 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:42:36.686 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:09:34 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T16:42:36.841 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T16:42:36.842 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:42:36.842 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-25T16:42:36.842 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:42:36.842 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T16:42:36.842 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:42:36.842 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:42:36.842 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T16:42:36.842 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T16:42:36.842 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-25T16:42:36.842 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T16:42:36.842 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 416 M 2024-03-25T16:42:36.843 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 1.7 G 2024-03-25T16:42:36.843 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T16:44:29.760 INFO:teuthology.orchestra.run.smithi016.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 272 kB/s | 68 MB 04:16 2024-03-25T16:44:29.761 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:44:29.761 INFO:teuthology.orchestra.run.smithi016.stdout:Total 362 kB/s | 91 MB 04:16 2024-03-25T16:44:29.761 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T16:44:29.766 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T16:44:29.766 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T16:44:30.376 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T16:44:30.376 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T16:44:33.648 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T16:44:41.158 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:44:41.229 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:44:44.569 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:44:44.569 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:44:44.834 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:44:44.834 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:44:44.834 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T16:44:44.834 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:44:44.834 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:44:44.834 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:44:44.834 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T16:44:45.048 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:44:45.624 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:11:26 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T16:44:45.779 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T16:44:45.780 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:44:45.780 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-03-25T16:44:45.780 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:44:45.780 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T16:44:45.780 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:44:45.780 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:44:45.780 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T16:44:45.780 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T16:44:45.780 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-25T16:44:45.780 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T16:44:45.780 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 416 M 2024-03-25T16:44:45.780 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 1.7 G 2024-03-25T16:44:45.781 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T17:01:30.807 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 424 kB/s | 416 MB 16:45 2024-03-25T17:01:30.808 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:01:30.808 INFO:teuthology.orchestra.run.smithi016.stdout:Total 424 kB/s | 416 MB 16:45 2024-03-25T17:01:30.808 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T17:01:30.810 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T17:01:30.810 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T17:01:33.399 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T17:01:33.399 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T17:02:17.960 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T17:02:18.063 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:02:23.235 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:02:23.619 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:02:23.619 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T17:02:23.620 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T17:02:23.620 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:02:23.620 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T17:02:23.620 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T17:02:23.864 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T17:02:24.440 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:29:05 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T17:02:24.595 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T17:02:24.595 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================== 2024-03-25T17:02:24.595 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-03-25T17:02:24.596 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================== 2024-03-25T17:02:24.596 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T17:02:24.596 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T17:02:24.596 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T17:02:24.596 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T17:02:24.596 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================== 2024-03-25T17:02:24.596 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-25T17:02:24.596 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T17:02:24.597 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 1.4 M 2024-03-25T17:02:24.597 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 6.4 M 2024-03-25T17:02:24.597 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T17:02:30.622 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 242 kB/s | 1.4 MB 00:06 2024-03-25T17:02:30.623 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:02:30.623 INFO:teuthology.orchestra.run.smithi016.stdout:Total 242 kB/s | 1.4 MB 00:06 2024-03-25T17:02:30.624 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T17:02:30.628 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T17:02:30.628 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T17:02:30.675 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T17:02:30.675 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T17:02:30.938 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T17:02:31.040 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:02:31.429 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:02:31.695 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:02:31.695 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T17:02:31.695 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T17:02:31.695 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:02:31.695 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T17:02:31.695 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T17:02:31.885 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T17:02:32.462 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:29:13 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T17:02:32.617 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T17:02:32.618 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T17:02:32.618 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-03-25T17:02:32.618 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T17:02:32.618 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T17:02:32.619 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T17:02:32.619 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T17:02:32.619 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T17:02:32.619 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T17:02:32.619 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-25T17:02:32.619 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T17:02:32.619 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 176 M 2024-03-25T17:02:32.619 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 753 M 2024-03-25T17:02:32.619 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T17:08:26.512 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 275 kB/s | 416 MB 25:49 2024-03-25T17:08:26.513 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:08:26.513 INFO:teuthology.orchestra.run.smithi003.stdout:Total 275 kB/s | 416 MB 25:49 2024-03-25T17:08:26.513 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T17:08:26.517 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T17:08:26.517 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T17:08:27.778 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 506 kB/s | 176 MB 05:55 2024-03-25T17:08:27.778 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:08:27.779 INFO:teuthology.orchestra.run.smithi016.stdout:Total 506 kB/s | 176 MB 05:55 2024-03-25T17:08:27.779 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T17:08:27.782 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T17:08:27.782 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T17:08:28.893 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T17:08:28.893 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T17:08:29.102 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T17:08:29.103 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T17:08:47.987 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T17:08:48.087 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:08:52.980 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:08:53.242 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:08:53.242 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T17:08:53.242 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T17:08:53.242 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:08:53.242 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T17:08:53.243 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T17:08:53.450 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:08:54.026 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:35:35 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T17:08:54.181 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T17:08:54.182 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T17:08:54.182 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-03-25T17:08:54.182 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T17:08:54.182 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T17:08:54.182 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:08:54.182 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T17:08:54.182 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T17:08:54.182 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T17:08:54.183 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-25T17:08:54.183 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T17:08:54.183 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 672 M 2024-03-25T17:08:54.183 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 3.1 G 2024-03-25T17:08:54.183 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T17:09:13.714 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T17:09:13.817 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:09:21.710 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:09:21.967 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:09:21.967 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T17:09:21.967 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T17:09:21.967 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:09:21.967 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T17:09:21.967 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T17:09:22.221 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T17:09:22.799 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:36:20 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T17:09:22.952 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T17:09:22.953 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2024-03-25T17:09:22.953 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-25T17:09:22.953 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2024-03-25T17:09:22.953 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T17:09:22.953 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T17:09:22.953 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T17:09:22.953 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T17:09:22.953 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2024-03-25T17:09:22.953 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-25T17:09:22.953 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T17:09:22.954 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 1.4 M 2024-03-25T17:09:22.954 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 6.4 M 2024-03-25T17:09:22.954 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T17:09:30.441 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 195 kB/s | 1.4 MB 00:07 2024-03-25T17:09:30.442 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:09:30.443 INFO:teuthology.orchestra.run.smithi003.stdout:Total 195 kB/s | 1.4 MB 00:07 2024-03-25T17:09:30.443 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T17:09:30.446 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T17:09:30.447 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T17:09:30.493 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T17:09:30.493 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T17:09:30.758 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T17:09:30.867 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:09:31.327 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:09:31.597 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:09:31.597 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T17:09:31.598 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T17:09:31.598 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:09:31.598 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T17:09:31.598 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T17:09:31.781 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T17:09:32.355 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:36:30 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T17:09:32.509 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T17:09:32.510 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T17:09:32.510 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-25T17:09:32.510 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T17:09:32.510 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T17:09:32.510 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T17:09:32.510 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T17:09:32.510 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T17:09:32.511 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T17:09:32.511 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-25T17:09:32.511 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T17:09:32.511 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 176 M 2024-03-25T17:09:32.511 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 753 M 2024-03-25T17:09:32.511 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T17:20:47.786 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 266 kB/s | 176 MB 11:15 2024-03-25T17:20:47.804 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:20:47.804 INFO:teuthology.orchestra.run.smithi003.stdout:Total 266 kB/s | 176 MB 11:15 2024-03-25T17:20:47.804 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T17:20:47.804 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T17:20:47.804 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T17:20:48.900 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T17:20:48.900 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T17:21:07.941 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T17:21:08.034 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:21:12.641 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:21:12.915 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:21:12.916 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T17:21:12.916 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T17:21:12.916 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:21:12.916 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T17:21:12.916 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T17:21:13.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:21:13.685 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:48:11 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T17:21:13.840 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T17:21:13.841 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T17:21:13.841 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-25T17:21:13.841 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T17:21:13.841 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T17:21:13.841 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:21:13.841 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T17:21:13.841 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T17:21:13.842 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T17:21:13.842 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-25T17:21:13.842 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T17:21:13.842 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 672 M 2024-03-25T17:21:13.842 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 3.1 G 2024-03-25T17:21:13.842 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T17:49:57.376 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 279 kB/s | 672 MB 41:03 2024-03-25T17:49:57.392 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:49:57.392 INFO:teuthology.orchestra.run.smithi016.stdout:Total 279 kB/s | 672 MB 41:03 2024-03-25T17:49:57.392 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T17:49:57.393 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T17:49:57.393 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T17:50:01.542 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T17:50:01.542 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T17:51:16.392 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T17:51:16.942 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:51:25.208 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:51:25.466 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:51:25.466 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T17:51:25.466 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T17:51:25.466 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:25.466 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T17:51:25.466 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T17:51:25.768 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:51:26.347 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 1:18:07 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T17:51:26.447 INFO:teuthology.orchestra.run.smithi016.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:51:26.502 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T17:51:26.503 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-03-25T17:51:26.503 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T17:51:26.580 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:51:27.157 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 1:18:08 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T17:51:27.310 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T17:51:27.311 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T17:51:27.311 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-03-25T17:51:27.311 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T17:51:27.311 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T17:51:27.311 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:51:27.311 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T17:51:27.311 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T17:51:27.311 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T17:51:27.312 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-25T17:51:27.312 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T17:51:27.312 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 20 M 2024-03-25T17:51:27.312 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 103 M 2024-03-25T17:51:27.312 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T17:52:32.852 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 308 kB/s | 20 MB 01:05 2024-03-25T17:52:32.852 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:32.852 INFO:teuthology.orchestra.run.smithi016.stdout:Total 308 kB/s | 20 MB 01:05 2024-03-25T17:52:32.852 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T17:52:32.856 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T17:52:32.856 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T17:52:33.011 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T17:52:33.012 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T17:52:35.367 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T17:52:35.484 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:52:36.479 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:52:36.746 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:52:36.746 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T17:52:36.746 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T17:52:36.746 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:52:36.746 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T17:52:36.746 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T17:52:36.940 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:52:37.516 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 1:19:18 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T17:52:37.672 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T17:52:37.673 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T17:52:37.673 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-03-25T17:52:37.674 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T17:52:37.674 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T17:52:37.674 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:52:37.674 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T17:52:37.674 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T17:52:37.674 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T17:52:37.674 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-25T17:52:37.674 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T17:52:37.674 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 48 M 2024-03-25T17:52:37.674 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 173 M 2024-03-25T17:52:37.674 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T17:53:38.736 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 354 kB/s | 672 MB 32:24 2024-03-25T17:53:38.737 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:53:38.737 INFO:teuthology.orchestra.run.smithi003.stdout:Total 354 kB/s | 672 MB 32:24 2024-03-25T17:53:38.737 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T17:53:38.742 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T17:53:38.742 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T17:53:42.894 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T17:53:42.894 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T17:54:57.776 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T17:54:57.880 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:55:05.326 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:55:05.590 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:55:05.590 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T17:55:05.590 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T17:55:05.590 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:05.590 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T17:55:05.590 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T17:55:05.867 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:55:06.444 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 1:22:04 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T17:55:06.542 INFO:teuthology.orchestra.run.smithi003.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:55:06.597 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T17:55:06.598 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-25T17:55:06.598 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T17:55:06.685 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:55:07.260 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 1:22:05 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T17:55:07.414 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T17:55:07.415 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T17:55:07.415 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-25T17:55:07.415 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T17:55:07.415 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T17:55:07.415 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:55:07.415 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T17:55:07.415 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T17:55:07.415 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T17:55:07.415 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-25T17:55:07.415 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T17:55:07.415 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 20 M 2024-03-25T17:55:07.415 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 103 M 2024-03-25T17:55:07.415 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T17:55:24.519 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 293 kB/s | 48 MB 02:46 2024-03-25T17:55:24.520 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:55:24.520 INFO:teuthology.orchestra.run.smithi016.stdout:Total 293 kB/s | 48 MB 02:46 2024-03-25T17:55:24.520 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T17:55:24.523 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T17:55:24.523 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T17:55:24.850 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T17:55:24.850 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T17:55:29.861 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T17:55:29.952 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:55:31.336 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:55:31.601 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:55:31.601 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T17:55:31.601 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T17:55:31.601 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:31.601 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T17:55:31.601 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T17:55:31.783 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:55:32.358 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 1:22:13 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T17:55:32.514 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T17:55:32.538 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T17:55:32.538 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-03-25T17:55:32.538 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T17:55:32.539 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T17:55:32.539 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:55:32.539 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T17:55:32.539 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T17:55:32.539 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T17:55:32.539 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-25T17:55:32.539 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T17:55:32.539 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 99 M 2024-03-25T17:55:32.539 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 449 M 2024-03-25T17:55:32.539 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T17:56:09.440 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 325 kB/s | 20 MB 01:02 2024-03-25T17:56:09.441 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:56:09.441 INFO:teuthology.orchestra.run.smithi003.stdout:Total 325 kB/s | 20 MB 01:02 2024-03-25T17:56:09.441 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T17:56:09.445 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T17:56:09.445 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T17:56:09.601 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T17:56:09.601 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T17:56:11.982 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T17:56:12.081 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:56:12.997 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:56:13.271 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:56:13.271 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T17:56:13.271 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T17:56:13.272 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:56:13.272 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T17:56:13.272 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T17:56:13.454 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:56:14.029 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 1:23:12 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T17:56:14.183 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T17:56:14.184 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T17:56:14.184 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-25T17:56:14.184 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T17:56:14.184 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T17:56:14.184 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:56:14.184 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T17:56:14.184 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T17:56:14.184 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T17:56:14.185 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-25T17:56:14.185 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T17:56:14.185 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 48 M 2024-03-25T17:56:14.185 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 173 M 2024-03-25T17:56:14.185 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T17:58:21.533 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 384 kB/s | 48 MB 02:07 2024-03-25T17:58:21.534 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:21.534 INFO:teuthology.orchestra.run.smithi003.stdout:Total 384 kB/s | 48 MB 02:07 2024-03-25T17:58:21.534 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T17:58:21.537 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T17:58:21.537 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T17:58:21.864 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T17:58:21.864 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T17:58:26.878 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T17:58:26.975 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:28.322 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:28.594 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:28.594 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T17:58:28.595 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T17:58:28.595 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:28.595 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T17:58:28.595 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T17:58:28.782 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:58:29.357 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 1:25:27 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T17:58:29.510 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T17:58:29.511 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T17:58:29.511 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-25T17:58:29.511 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T17:58:29.511 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T17:58:29.511 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:58:29.511 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T17:58:29.511 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T17:58:29.512 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T17:58:29.512 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-25T17:58:29.512 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T17:58:29.512 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 99 M 2024-03-25T17:58:29.512 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 449 M 2024-03-25T17:58:29.512 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T18:00:20.941 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 350 kB/s | 99 MB 04:48 2024-03-25T18:00:20.941 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:00:20.942 INFO:teuthology.orchestra.run.smithi016.stdout:Total 350 kB/s | 99 MB 04:48 2024-03-25T18:00:20.942 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T18:00:20.944 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T18:00:20.944 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T18:00:21.583 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T18:00:21.583 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T18:00:32.482 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T18:00:32.582 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:00:35.567 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:00:35.838 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:00:35.839 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:00:35.839 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T18:00:35.839 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:35.839 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:00:35.839 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T18:00:36.046 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T18:00:36.620 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 1:27:17 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T18:00:36.775 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T18:00:36.776 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:00:36.776 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-03-25T18:00:36.776 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:00:36.776 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T18:00:36.776 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T18:00:36.776 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:00:36.776 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T18:00:36.776 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:00:36.777 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-25T18:00:36.777 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:00:36.777 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 287 M 2024-03-25T18:00:36.777 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 1.2 G 2024-03-25T18:00:36.777 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T18:04:07.356 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 299 kB/s | 99 MB 05:37 2024-03-25T18:04:07.358 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:07.358 INFO:teuthology.orchestra.run.smithi003.stdout:Total 299 kB/s | 99 MB 05:37 2024-03-25T18:04:07.358 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T18:04:07.360 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T18:04:07.360 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T18:04:07.997 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T18:04:07.998 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T18:04:18.870 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T18:04:18.980 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:04:21.818 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:04:22.083 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:04:22.083 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:04:22.083 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T18:04:22.083 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:22.083 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:04:22.083 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T18:04:22.261 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T18:04:22.836 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 1:31:20 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T18:04:22.989 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T18:04:22.990 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:04:22.990 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-25T18:04:22.990 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:04:22.991 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T18:04:22.991 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T18:04:22.991 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:04:22.991 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T18:04:22.991 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:04:22.991 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-25T18:04:22.991 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:04:22.991 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 287 M 2024-03-25T18:04:22.991 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 1.2 G 2024-03-25T18:04:22.991 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T18:15:18.506 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 333 kB/s | 287 MB 14:41 2024-03-25T18:15:18.507 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:15:18.507 INFO:teuthology.orchestra.run.smithi016.stdout:Total 333 kB/s | 287 MB 14:41 2024-03-25T18:15:18.507 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T18:15:18.510 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T18:15:18.510 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T18:15:20.299 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T18:15:20.300 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T18:15:51.400 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T18:15:51.504 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:15:59.201 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:15:59.493 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:15:59.493 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:15:59.493 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T18:15:59.493 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:15:59.493 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:15:59.493 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T18:15:59.727 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T18:16:00.305 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 1:42:41 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T18:16:00.459 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T18:16:00.460 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:16:00.460 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-03-25T18:16:00.460 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:16:00.460 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T18:16:00.460 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T18:16:00.460 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:16:00.460 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T18:16:00.461 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:16:00.461 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-25T18:16:00.461 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:16:00.461 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 5.9 M 2024-03-25T18:16:00.461 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 28 M 2024-03-25T18:16:00.461 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T18:16:12.429 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 508 kB/s | 5.9 MB 00:11 2024-03-25T18:16:12.430 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:16:12.430 INFO:teuthology.orchestra.run.smithi016.stdout:Total 508 kB/s | 5.9 MB 00:11 2024-03-25T18:16:12.431 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T18:16:12.435 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T18:16:12.435 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T18:16:12.509 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T18:16:12.510 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T18:16:13.301 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T18:16:13.413 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:16:14.008 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:16:14.285 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:16:14.285 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:16:14.285 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T18:16:14.286 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:16:14.286 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:16:14.286 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T18:16:14.488 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install librados-devel-debuginfo 2024-03-25T18:16:15.066 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 1:42:56 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T18:16:15.221 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T18:16:15.222 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:16:15.222 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-03-25T18:16:15.222 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:16:15.222 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T18:16:15.222 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T18:16:15.222 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:16:15.222 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T18:16:15.223 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:16:15.223 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-25T18:16:15.223 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:16:15.223 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 109 k 2024-03-25T18:16:15.223 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 440 k 2024-03-25T18:16:15.223 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T18:16:15.932 INFO:teuthology.orchestra.run.smithi016.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 153 kB/s | 109 kB 00:00 2024-03-25T18:16:15.933 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:16:15.933 INFO:teuthology.orchestra.run.smithi016.stdout:Total 153 kB/s | 109 kB 00:00 2024-03-25T18:16:15.933 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T18:16:15.936 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T18:16:15.937 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T18:16:15.972 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T18:16:15.973 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T18:16:16.091 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T18:16:16.209 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:16:16.593 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:16:16.858 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:16:16.859 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:16:16.859 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T18:16:16.859 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:16:16.859 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:16:16.859 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T18:16:17.058 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T18:16:17.635 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 1:42:58 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T18:16:17.789 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T18:16:17.789 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:16:17.789 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-03-25T18:16:17.789 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:16:17.790 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T18:16:17.790 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T18:16:17.790 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:16:17.790 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T18:16:17.790 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:16:17.790 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-25T18:16:17.790 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:16:17.790 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 5.4 M 2024-03-25T18:16:17.791 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 25 M 2024-03-25T18:16:17.791 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T18:16:26.985 INFO:teuthology.orchestra.run.smithi016.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 596 kB/s | 5.4 MB 00:09 2024-03-25T18:16:26.985 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:16:26.986 INFO:teuthology.orchestra.run.smithi016.stdout:Total 596 kB/s | 5.4 MB 00:09 2024-03-25T18:16:26.986 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T18:16:26.990 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T18:16:26.990 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T18:16:27.062 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T18:16:27.062 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T18:16:27.836 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T18:16:27.936 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:16:28.472 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:16:28.741 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:16:28.741 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:16:28.741 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T18:16:28.741 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:16:28.741 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:16:28.741 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T18:16:28.918 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install librados2-debuginfo 2024-03-25T18:16:29.496 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 1:43:10 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T18:16:29.650 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T18:16:29.651 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:16:29.651 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-03-25T18:16:29.651 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:16:29.651 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T18:16:29.651 INFO:teuthology.orchestra.run.smithi016.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T18:16:29.651 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:16:29.651 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T18:16:29.651 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:16:29.651 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-25T18:16:29.651 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:16:29.652 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 44 M 2024-03-25T18:16:29.652 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 253 M 2024-03-25T18:16:29.652 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T18:17:26.686 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 375 kB/s | 287 MB 13:03 2024-03-25T18:17:26.686 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:17:26.686 INFO:teuthology.orchestra.run.smithi003.stdout:Total 375 kB/s | 287 MB 13:03 2024-03-25T18:17:26.687 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T18:17:26.689 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T18:17:26.689 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T18:17:28.479 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T18:17:28.479 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T18:17:59.548 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T18:17:59.649 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:18:06.765 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:18:07.053 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:18:07.053 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:18:07.053 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T18:18:07.053 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:07.053 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:18:07.053 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T18:18:07.259 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T18:18:07.833 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 1:45:05 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T18:18:07.988 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T18:18:07.989 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:18:07.989 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-25T18:18:07.989 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:18:07.989 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T18:18:07.989 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T18:18:07.989 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:18:07.989 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T18:18:07.989 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:18:07.989 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-25T18:18:07.989 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:18:07.990 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 5.9 M 2024-03-25T18:18:07.990 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 28 M 2024-03-25T18:18:07.990 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T18:18:20.457 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 488 kB/s | 5.9 MB 00:12 2024-03-25T18:18:20.458 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:20.458 INFO:teuthology.orchestra.run.smithi003.stdout:Total 488 kB/s | 5.9 MB 00:12 2024-03-25T18:18:20.458 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T18:18:20.462 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T18:18:20.462 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T18:18:20.536 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T18:18:20.537 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T18:18:21.307 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T18:18:21.408 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:18:22.000 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:18:22.269 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:18:22.269 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:18:22.269 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T18:18:22.269 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:22.269 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:18:22.269 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T18:18:22.463 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librados-devel-debuginfo 2024-03-25T18:18:23.036 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 1:45:21 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T18:18:23.191 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T18:18:23.192 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:18:23.192 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-25T18:18:23.192 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:18:23.192 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T18:18:23.192 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T18:18:23.192 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:18:23.192 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T18:18:23.192 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:18:23.193 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-25T18:18:23.193 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:18:23.193 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 109 k 2024-03-25T18:18:23.193 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 440 k 2024-03-25T18:18:23.193 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T18:18:24.134 INFO:teuthology.orchestra.run.smithi016.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 396 kB/s | 44 MB 01:54 2024-03-25T18:18:24.134 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:24.135 INFO:teuthology.orchestra.run.smithi016.stdout:Total 396 kB/s | 44 MB 01:54 2024-03-25T18:18:24.135 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T18:18:24.139 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T18:18:24.139 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T18:18:24.301 INFO:teuthology.orchestra.run.smithi003.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 98 kB/s | 109 kB 00:01 2024-03-25T18:18:24.302 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:24.302 INFO:teuthology.orchestra.run.smithi003.stdout:Total 98 kB/s | 109 kB 00:01 2024-03-25T18:18:24.302 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T18:18:24.306 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T18:18:24.306 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T18:18:24.345 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T18:18:24.345 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T18:18:24.453 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T18:18:24.453 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T18:18:24.471 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T18:18:24.589 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:18:25.017 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:18:25.294 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:18:25.294 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:18:25.294 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T18:18:25.294 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:25.294 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:18:25.294 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T18:18:25.499 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T18:18:26.077 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 1:45:24 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T18:18:26.229 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T18:18:26.230 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:18:26.230 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-25T18:18:26.230 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:18:26.230 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T18:18:26.230 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T18:18:26.230 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:18:26.230 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T18:18:26.230 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:18:26.230 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-25T18:18:26.230 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:18:26.231 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 5.4 M 2024-03-25T18:18:26.231 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 25 M 2024-03-25T18:18:26.231 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T18:18:29.826 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T18:18:29.952 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:18:31.845 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:18:32.107 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:18:32.107 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:18:32.107 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T18:18:32.107 INFO:teuthology.orchestra.run.smithi016.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:32.107 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:18:32.107 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T18:18:32.297 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:18:32.875 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 1:45:13 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T18:18:33.031 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T18:18:33.032 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:18:33.032 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-03-25T18:18:33.032 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:18:33.032 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T18:18:33.032 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:18:33.032 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:18:33.032 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T18:18:33.032 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:18:33.032 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-25T18:18:33.032 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:18:33.033 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 52 M 2024-03-25T18:18:33.033 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 308 M 2024-03-25T18:18:33.033 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T18:18:35.520 INFO:teuthology.orchestra.run.smithi003.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 590 kB/s | 5.4 MB 00:09 2024-03-25T18:18:35.521 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:35.521 INFO:teuthology.orchestra.run.smithi003.stdout:Total 590 kB/s | 5.4 MB 00:09 2024-03-25T18:18:35.522 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T18:18:35.526 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T18:18:35.526 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T18:18:35.597 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T18:18:35.597 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T18:18:36.294 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T18:18:36.405 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:18:37.088 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:18:37.358 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:18:37.358 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:18:37.359 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T18:18:37.359 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:37.359 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:18:37.359 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T18:18:37.558 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librados2-debuginfo 2024-03-25T18:18:38.133 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 1:45:36 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T18:18:38.287 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T18:18:38.288 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:18:38.288 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-25T18:18:38.288 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:18:38.288 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T18:18:38.288 INFO:teuthology.orchestra.run.smithi003.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T18:18:38.288 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:18:38.288 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T18:18:38.288 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:18:38.288 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-25T18:18:38.289 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:18:38.289 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 44 M 2024-03-25T18:18:38.289 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 253 M 2024-03-25T18:18:38.289 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T18:20:25.319 INFO:teuthology.orchestra.run.smithi016.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 473 kB/s | 52 MB 01:52 2024-03-25T18:20:25.319 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:20:25.319 INFO:teuthology.orchestra.run.smithi016.stdout:Total 473 kB/s | 52 MB 01:52 2024-03-25T18:20:25.319 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T18:20:25.323 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T18:20:25.323 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T18:20:25.682 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T18:20:25.682 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T18:20:30.676 INFO:teuthology.orchestra.run.smithi003.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 403 kB/s | 44 MB 01:52 2024-03-25T18:20:30.677 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:20:30.677 INFO:teuthology.orchestra.run.smithi003.stdout:Total 403 kB/s | 44 MB 01:52 2024-03-25T18:20:30.677 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T18:20:30.680 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T18:20:30.680 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T18:20:30.993 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T18:20:30.993 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T18:20:32.018 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T18:20:32.111 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:20:34.282 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:20:34.681 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:20:34.681 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:20:34.681 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T18:20:34.681 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:34.681 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:20:34.681 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T18:20:34.866 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:20:35.434 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 1:47:16 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T18:20:35.589 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T18:20:35.590 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:20:35.590 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-03-25T18:20:35.590 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:20:35.590 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T18:20:35.590 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:20:35.590 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:20:35.590 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T18:20:35.590 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:20:35.591 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-25T18:20:35.591 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:20:35.591 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 615 k 2024-03-25T18:20:35.591 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 2.1 M 2024-03-25T18:20:35.591 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T18:20:36.353 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T18:20:36.460 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:20:38.358 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:20:38.431 INFO:teuthology.orchestra.run.smithi016.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 216 kB/s | 615 kB 00:02 2024-03-25T18:20:38.432 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:20:38.432 INFO:teuthology.orchestra.run.smithi016.stdout:Total 216 kB/s | 615 kB 00:02 2024-03-25T18:20:38.432 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T18:20:38.437 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T18:20:38.437 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T18:20:38.479 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T18:20:38.480 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T18:20:38.651 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T18:20:38.663 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:20:38.664 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:20:38.664 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T18:20:38.664 INFO:teuthology.orchestra.run.smithi003.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:38.664 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:20:38.664 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T18:20:38.766 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:20:38.861 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:20:39.138 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:20:39.396 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:20:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:20:39.397 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T18:20:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:20:39.397 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T18:20:39.434 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 1:47:37 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T18:20:39.568 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:20:39.591 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T18:20:39.591 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:20:39.592 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-25T18:20:39.592 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:20:39.592 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T18:20:39.592 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:20:39.592 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:20:39.592 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T18:20:39.592 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:20:39.592 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-25T18:20:39.592 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:20:39.593 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 52 M 2024-03-25T18:20:39.593 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 308 M 2024-03-25T18:20:39.593 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T18:20:40.141 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 1:47:21 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T18:20:40.297 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T18:20:40.298 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:20:40.298 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-03-25T18:20:40.298 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:20:40.299 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T18:20:40.299 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:20:40.299 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:20:40.299 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T18:20:40.299 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:20:40.299 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-25T18:20:40.299 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:20:40.299 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 771 k 2024-03-25T18:20:40.299 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 2.7 M 2024-03-25T18:20:40.299 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T18:20:43.875 INFO:teuthology.orchestra.run.smithi016.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 216 kB/s | 771 kB 00:03 2024-03-25T18:20:43.876 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:20:43.876 INFO:teuthology.orchestra.run.smithi016.stdout:Total 216 kB/s | 771 kB 00:03 2024-03-25T18:20:43.876 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T18:20:43.880 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T18:20:43.880 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T18:20:43.924 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T18:20:43.924 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T18:20:44.119 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T18:20:44.228 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:20:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:20:44.880 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:20:44.880 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:20:44.881 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T18:20:44.881 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:44.881 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:20:44.881 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T18:20:45.068 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:20:45.639 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 1:47:26 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T18:20:45.795 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T18:20:45.795 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:20:45.796 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-03-25T18:20:45.796 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:20:45.796 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T18:20:45.796 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:20:45.796 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:20:45.796 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T18:20:45.796 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:20:45.796 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-25T18:20:45.796 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:20:45.797 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 1.1 M 2024-03-25T18:20:45.797 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 4.3 M 2024-03-25T18:20:45.797 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T18:20:52.511 INFO:teuthology.orchestra.run.smithi016.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 171 kB/s | 1.1 MB 00:06 2024-03-25T18:20:52.512 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:20:52.512 INFO:teuthology.orchestra.run.smithi016.stdout:Total 171 kB/s | 1.1 MB 00:06 2024-03-25T18:20:52.512 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T18:20:52.517 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T18:20:52.517 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T18:20:52.562 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T18:20:52.562 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T18:20:52.791 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T18:20:52.896 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:20:53.287 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:20:53.547 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:20:53.547 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:20:53.547 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T18:20:53.547 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:53.547 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:20:53.548 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T18:20:53.774 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:20:54.349 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 1:47:35 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T18:20:54.506 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T18:20:54.507 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:20:54.507 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-03-25T18:20:54.507 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:20:54.507 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T18:20:54.507 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:20:54.507 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:20:54.507 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T18:20:54.507 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:20:54.507 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-25T18:20:54.507 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:20:54.507 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 337 k 2024-03-25T18:20:54.507 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 1.0 M 2024-03-25T18:20:54.507 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T18:20:56.064 INFO:teuthology.orchestra.run.smithi016.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 217 kB/s | 337 kB 00:01 2024-03-25T18:20:56.065 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:20:56.065 INFO:teuthology.orchestra.run.smithi016.stdout:Total 216 kB/s | 337 kB 00:01 2024-03-25T18:20:56.065 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T18:20:56.069 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T18:20:56.069 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T18:20:56.110 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T18:20:56.110 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T18:20:56.255 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T18:20:56.358 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:20:56.773 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:20:57.177 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:20:57.177 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:20:57.177 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T18:20:57.178 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:57.178 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:20:57.178 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T18:20:57.369 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:20:57.944 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 1:47:38 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T18:20:58.098 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T18:20:58.099 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:20:58.099 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-03-25T18:20:58.099 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:20:58.099 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T18:20:58.099 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:20:58.099 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:20:58.099 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T18:20:58.099 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:20:58.099 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-25T18:20:58.099 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:20:58.100 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 509 k 2024-03-25T18:20:58.100 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 2.5 M 2024-03-25T18:20:58.100 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T18:20:59.900 INFO:teuthology.orchestra.run.smithi016.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 283 kB/s | 509 kB 00:01 2024-03-25T18:20:59.901 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:20:59.901 INFO:teuthology.orchestra.run.smithi016.stdout:Total 283 kB/s | 509 kB 00:01 2024-03-25T18:20:59.901 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T18:20:59.905 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T18:20:59.906 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T18:20:59.947 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T18:20:59.947 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T18:21:00.112 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T18:21:00.229 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:21:00.664 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:21:00.930 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:21:00.930 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:21:00.930 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T18:21:00.930 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:21:00.930 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:21:00.930 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T18:21:01.117 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:21:01.691 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 1:47:42 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T18:21:01.847 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T18:21:01.848 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:21:01.848 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-03-25T18:21:01.848 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:21:01.848 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T18:21:01.848 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:21:01.848 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:21:01.848 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T18:21:01.848 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:21:01.848 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-25T18:21:01.848 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:21:01.848 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 52 M 2024-03-25T18:21:01.848 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 298 M 2024-03-25T18:21:01.849 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T18:21:43.653 INFO:teuthology.orchestra.run.smithi003.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 829 kB/s | 52 MB 01:04 2024-03-25T18:21:43.654 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:21:43.654 INFO:teuthology.orchestra.run.smithi003.stdout:Total 829 kB/s | 52 MB 01:04 2024-03-25T18:21:43.654 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T18:21:43.658 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T18:21:43.658 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T18:21:44.017 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T18:21:44.017 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T18:21:50.355 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T18:21:50.453 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:21:52.627 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:21:52.903 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:21:52.904 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:21:52.904 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T18:21:52.904 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:21:52.904 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:21:52.904 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T18:21:53.110 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:21:53.678 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 1:48:51 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T18:21:53.832 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T18:21:53.833 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:21:53.833 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-25T18:21:53.833 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:21:53.833 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T18:21:53.833 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:21:53.833 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:21:53.833 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T18:21:53.833 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:21:53.833 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-25T18:21:53.833 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:21:53.834 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 615 k 2024-03-25T18:21:53.834 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 2.1 M 2024-03-25T18:21:53.834 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T18:21:55.972 INFO:teuthology.orchestra.run.smithi003.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 288 kB/s | 615 kB 00:02 2024-03-25T18:21:55.972 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:21:55.973 INFO:teuthology.orchestra.run.smithi003.stdout:Total 287 kB/s | 615 kB 00:02 2024-03-25T18:21:55.973 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T18:21:55.977 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T18:21:55.977 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T18:21:56.021 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T18:21:56.021 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T18:21:56.197 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T18:21:56.307 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:21:56.704 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:21:56.976 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:21:56.976 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:21:56.976 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T18:21:56.976 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:21:56.976 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:21:56.976 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T18:21:57.177 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:21:57.748 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 1:48:55 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T18:21:57.904 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T18:21:57.904 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:21:57.904 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-25T18:21:57.904 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:21:57.904 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T18:21:57.904 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:21:57.905 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:21:57.905 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T18:21:57.905 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:21:57.905 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-25T18:21:57.905 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:21:57.905 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 771 k 2024-03-25T18:21:57.905 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 2.7 M 2024-03-25T18:21:57.906 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T18:22:00.966 INFO:teuthology.orchestra.run.smithi003.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 252 kB/s | 771 kB 00:03 2024-03-25T18:22:00.967 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:22:00.967 INFO:teuthology.orchestra.run.smithi003.stdout:Total 252 kB/s | 771 kB 00:03 2024-03-25T18:22:00.968 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T18:22:00.971 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T18:22:00.972 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T18:22:01.014 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T18:22:01.014 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T18:22:01.203 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T18:22:01.318 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:22:01.724 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:22:02.002 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:22:02.002 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:22:02.002 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T18:22:02.002 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:22:02.002 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:22:02.002 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T18:22:02.232 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:22:02.804 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 1:49:00 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T18:22:02.961 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T18:22:02.962 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:22:02.962 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-25T18:22:02.962 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:22:02.962 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T18:22:02.962 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:22:02.962 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:22:02.962 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T18:22:02.962 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:22:02.963 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-25T18:22:02.963 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:22:02.963 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 1.1 M 2024-03-25T18:22:02.963 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 4.3 M 2024-03-25T18:22:02.963 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T18:22:04.886 INFO:teuthology.orchestra.run.smithi003.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 598 kB/s | 1.1 MB 00:01 2024-03-25T18:22:04.887 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:22:04.887 INFO:teuthology.orchestra.run.smithi003.stdout:Total 598 kB/s | 1.1 MB 00:01 2024-03-25T18:22:04.888 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T18:22:04.892 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T18:22:04.892 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T18:22:04.936 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T18:22:04.937 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T18:22:05.168 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T18:22:05.272 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:22:05.692 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:22:05.968 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:22:05.968 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:22:05.968 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T18:22:05.968 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:22:05.968 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:22:05.968 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T18:22:06.212 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:22:06.787 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 1:49:04 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T18:22:06.946 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T18:22:06.946 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:22:06.946 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-25T18:22:06.946 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:22:06.947 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T18:22:06.947 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:22:06.947 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:22:06.947 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T18:22:06.947 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:22:06.947 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-25T18:22:06.947 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:22:06.948 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 337 k 2024-03-25T18:22:06.948 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 1.0 M 2024-03-25T18:22:06.948 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T18:22:07.914 INFO:teuthology.orchestra.run.smithi003.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 349 kB/s | 337 kB 00:00 2024-03-25T18:22:07.915 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:22:07.915 INFO:teuthology.orchestra.run.smithi003.stdout:Total 348 kB/s | 337 kB 00:00 2024-03-25T18:22:07.915 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T18:22:07.919 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T18:22:07.919 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T18:22:07.961 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T18:22:07.961 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T18:22:08.107 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T18:22:08.208 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:22:08.605 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:22:08.866 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:22:08.866 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:22:08.866 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T18:22:08.866 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:22:08.866 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:22:08.867 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T18:22:09.087 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:22:09.656 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 1:49:07 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T18:22:09.810 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T18:22:09.811 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:22:09.811 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-25T18:22:09.811 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:22:09.811 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T18:22:09.811 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:22:09.811 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:22:09.811 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T18:22:09.811 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:22:09.811 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-25T18:22:09.811 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:22:09.812 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 509 k 2024-03-25T18:22:09.812 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 2.5 M 2024-03-25T18:22:09.812 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T18:22:11.384 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 324 kB/s | 509 kB 00:01 2024-03-25T18:22:11.385 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:22:11.385 INFO:teuthology.orchestra.run.smithi003.stdout:Total 323 kB/s | 509 kB 00:01 2024-03-25T18:22:11.385 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T18:22:11.389 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T18:22:11.389 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T18:22:11.429 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T18:22:11.429 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T18:22:11.594 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T18:22:11.710 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:22:12.136 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:22:12.516 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:22:12.516 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:22:12.516 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T18:22:12.516 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:22:12.516 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:22:12.516 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T18:22:12.739 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:22:13.311 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 1:49:11 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T18:22:13.466 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T18:22:13.467 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:22:13.467 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-25T18:22:13.467 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:22:13.467 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T18:22:13.467 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:22:13.467 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:22:13.467 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T18:22:13.467 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:22:13.467 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-25T18:22:13.467 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:22:13.468 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 52 M 2024-03-25T18:22:13.468 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 298 M 2024-03-25T18:22:13.468 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T18:23:09.791 INFO:teuthology.orchestra.run.smithi016.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 413 kB/s | 52 MB 02:07 2024-03-25T18:23:09.792 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:09.792 INFO:teuthology.orchestra.run.smithi016.stdout:Total 413 kB/s | 52 MB 02:07 2024-03-25T18:23:09.792 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T18:23:09.795 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T18:23:09.795 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T18:23:10.146 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T18:23:10.146 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T18:23:16.391 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T18:23:16.491 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:23:18.577 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:23:18.834 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:23:18.834 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:23:18.834 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T18:23:18.834 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:18.834 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:23:18.834 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T18:23:19.043 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:23:19.625 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 1:50:00 ago on Mon 25 Mar 2024 04:33:19 PM UTC. 2024-03-25T18:23:19.779 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-25T18:23:19.779 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:23:19.779 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-03-25T18:23:19.779 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:23:19.779 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-25T18:23:19.779 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:23:19.779 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:23:19.780 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-25T18:23:19.780 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-25T18:23:19.780 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-25T18:23:19.780 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:23:19.780 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 1.3 M 2024-03-25T18:23:19.780 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 5.5 M 2024-03-25T18:23:19.780 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-25T18:23:21.859 INFO:teuthology.orchestra.run.smithi016.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 617 kB/s | 1.3 MB 00:02 2024-03-25T18:23:21.859 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:21.860 INFO:teuthology.orchestra.run.smithi016.stdout:Total 616 kB/s | 1.3 MB 00:02 2024-03-25T18:23:21.860 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-25T18:23:21.864 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-25T18:23:21.864 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-25T18:23:21.910 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-25T18:23:21.910 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-25T18:23:22.157 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-25T18:23:22.269 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:23:22.694 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:23:22.960 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:23:22.961 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:23:22.961 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-25T18:23:22.961 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:22.961 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:23:22.961 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-25T18:23:23.171 DEBUG:teuthology.parallel:result is None 2024-03-25T18:23:59.081 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 500 kB/s | 52 MB 01:45 2024-03-25T18:23:59.082 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:59.082 INFO:teuthology.orchestra.run.smithi003.stdout:Total 500 kB/s | 52 MB 01:45 2024-03-25T18:23:59.082 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T18:23:59.086 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T18:23:59.086 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T18:23:59.438 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T18:23:59.438 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T18:24:05.681 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T18:24:05.790 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:24:07.820 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:24:08.090 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:24:08.090 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:08.090 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T18:24:08.090 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:24:08.090 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:08.090 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T18:24:08.297 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:24:08.874 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 1:51:06 ago on Mon 25 Mar 2024 04:33:02 PM UTC. 2024-03-25T18:24:09.029 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-25T18:24:09.029 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:24:09.030 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-25T18:24:09.030 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:24:09.030 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-25T18:24:09.030 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:24:09.030 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:09.030 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-25T18:24:09.030 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-25T18:24:09.030 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-25T18:24:09.030 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:09.031 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 1.3 M 2024-03-25T18:24:09.031 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 5.5 M 2024-03-25T18:24:09.031 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-25T18:24:10.867 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 698 kB/s | 1.3 MB 00:01 2024-03-25T18:24:10.867 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:24:10.868 INFO:teuthology.orchestra.run.smithi003.stdout:Total 698 kB/s | 1.3 MB 00:01 2024-03-25T18:24:10.868 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-25T18:24:10.872 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-25T18:24:10.872 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-25T18:24:10.919 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-25T18:24:10.919 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-25T18:24:11.167 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-25T18:24:11.277 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:11.668 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:11.940 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:11.940 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:11.940 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-25T18:24:11.941 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:24:11.941 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:11.941 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-25T18:24:12.145 DEBUG:teuthology.parallel:result is None 2024-03-25T18:24:12.146 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:24:12.298 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:24:12.319 INFO:teuthology.orchestra.run.smithi003.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:24:12.319 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:24:12.319 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:24:12.321 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:24:12.542 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:24:12.565 INFO:teuthology.orchestra.run.smithi016.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:24:12.566 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:24:12.566 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:24:12.567 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T18:24:12.568 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-25T18:24:12.568 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:24:12.595 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-25T18:24:12.595 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:24:12.637 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T18:24:12.637 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-25T18:24:12.637 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:24:12.665 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:24:12.731 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-25T18:24:12.732 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:24:12.762 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:24:12.833 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T18:24:12.833 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-25T18:24:12.833 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:24:12.861 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:24:12.928 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-25T18:24:12.928 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:24:12.959 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:24:13.028 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T18:24:13.028 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-25T18:24:13.028 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:24:13.055 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:24:13.123 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-25T18:24:13.123 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:24:13.151 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:24:13.220 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T18:24:13.299 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T18:24:13.299 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 777 /var/log/ceph 2024-03-25T18:24:13.301 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 777 /var/log/ceph 2024-03-25T18:24:13.330 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T18:24:13.330 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:24:13.369 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:24:13.398 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T18:24:13.398 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:24:13.436 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:24:13.491 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T18:24:13.491 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T18:24:13.491 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621281', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621281', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi003.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGb6LbRgpGpUFYcRAE4NBLu+f/g5s85+pn1GZ0fF1ALVY6cOt3bertjioP6NLkPBZ2n6pGSpneMDUmqNHXIINj0=', 'smithi016.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDZzY+f/rTpKVXTkcqZ4u8ZEZyO6gAkY6E41GQQjYvdPQO9C1pOLTahnFTuRxn6R+lK4BV9WlwaNA0pFgkOC2Gg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T18:24:13.492 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:24:13.504 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:24:13.520 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:24:13.559 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:24:13.588 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-25T18:24:13.588 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:24:13.638 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:24:13.638 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_1 2024-03-25T18:24:13.693 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:24:13.693 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:24:13.693 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-25T18:24:13.693 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:24:13.693 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:24:13.693 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-03-25 16:39:19.805574856 +0000 2024-03-25T18:24:13.693 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-03-25 16:32:13.401758002 +0000 2024-03-25T18:24:13.693 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-03-25 16:32:13.401758002 +0000 2024-03-25T18:24:13.694 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-03-25 16:32:13.401758002 +0000 2024-03-25T18:24:13.694 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:24:13.757 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-03-25T18:24:13.757 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-03-25T18:24:13.757 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000250297 s, 2.0 MB/s 2024-03-25T18:24:13.759 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:24:13.814 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_2 2024-03-25T18:24:13.867 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:24:13.868 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:24:13.868 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-25T18:24:13.868 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:24:13.868 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:24:13.868 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-03-25 16:39:19.806574842 +0000 2024-03-25T18:24:13.868 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-03-25 16:32:13.780752571 +0000 2024-03-25T18:24:13.868 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-03-25 16:32:13.780752571 +0000 2024-03-25T18:24:13.868 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-03-25 16:32:13.780752571 +0000 2024-03-25T18:24:13.868 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:24:13.932 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-03-25T18:24:13.932 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-03-25T18:24:13.932 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000196765 s, 2.6 MB/s 2024-03-25T18:24:13.933 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:24:13.988 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_3 2024-03-25T18:24:14.043 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:24:14.043 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:24:14.043 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-25T18:24:14.043 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:24:14.043 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:24:14.043 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-03-25 16:39:19.806574842 +0000 2024-03-25T18:24:14.043 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-03-25 16:32:14.114747784 +0000 2024-03-25T18:24:14.043 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-03-25 16:32:14.114747784 +0000 2024-03-25T18:24:14.043 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-03-25 16:32:14.114747784 +0000 2024-03-25T18:24:14.043 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:24:14.107 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-03-25T18:24:14.107 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-03-25T18:24:14.107 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000171587 s, 3.0 MB/s 2024-03-25T18:24:14.109 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:24:14.164 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_4 2024-03-25T18:24:14.218 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:24:14.218 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:24:14.218 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-25T18:24:14.218 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:24:14.218 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:24:14.218 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-03-25 16:39:19.807574828 +0000 2024-03-25T18:24:14.218 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-03-25 16:32:14.452742940 +0000 2024-03-25T18:24:14.219 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-03-25 16:32:14.452742940 +0000 2024-03-25T18:24:14.219 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-03-25 16:32:14.452742940 +0000 2024-03-25T18:24:14.219 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:24:14.283 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-03-25T18:24:14.283 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-03-25T18:24:14.283 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000160361 s, 3.2 MB/s 2024-03-25T18:24:14.284 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:24:14.339 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:24:14.339 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-25T18:24:14.340 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:24:14.357 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:24:14.358 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_1 2024-03-25T18:24:14.415 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:24:14.415 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:24:14.415 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-25T18:24:14.415 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:24:14.415 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:24:14.415 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-03-25 16:40:08.613740161 +0000 2024-03-25T18:24:14.415 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-03-25 16:32:13.306972045 +0000 2024-03-25T18:24:14.415 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-03-25 16:32:13.306972045 +0000 2024-03-25T18:24:14.415 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-03-25 16:32:13.306972045 +0000 2024-03-25T18:24:14.416 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:24:14.484 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-03-25T18:24:14.484 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-03-25T18:24:14.484 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.00011397 s, 4.5 MB/s 2024-03-25T18:24:14.485 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:24:14.543 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_2 2024-03-25T18:24:14.600 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:24:14.601 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:24:14.601 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-25T18:24:14.601 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:24:14.601 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:24:14.601 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-03-25 16:40:08.614740146 +0000 2024-03-25T18:24:14.601 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-03-25 16:32:13.638967003 +0000 2024-03-25T18:24:14.601 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-03-25 16:32:13.638967003 +0000 2024-03-25T18:24:14.601 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-03-25 16:32:13.638967003 +0000 2024-03-25T18:24:14.601 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:24:14.669 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-03-25T18:24:14.669 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-03-25T18:24:14.669 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000100939 s, 5.1 MB/s 2024-03-25T18:24:14.670 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:24:14.728 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_3 2024-03-25T18:24:14.784 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:24:14.785 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:24:14.785 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-25T18:24:14.785 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:24:14.785 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:24:14.785 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-03-25 16:40:08.614740146 +0000 2024-03-25T18:24:14.785 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-03-25 16:32:13.977961855 +0000 2024-03-25T18:24:14.785 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-03-25 16:32:13.977961855 +0000 2024-03-25T18:24:14.785 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-03-25 16:32:13.977961855 +0000 2024-03-25T18:24:14.785 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:24:14.852 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-03-25T18:24:14.852 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-03-25T18:24:14.852 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000100912 s, 5.1 MB/s 2024-03-25T18:24:14.853 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:24:14.911 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_4 2024-03-25T18:24:14.967 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:24:14.967 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:24:14.967 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-25T18:24:14.967 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:24:14.967 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:24:14.967 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-03-25 16:40:08.615740131 +0000 2024-03-25T18:24:14.967 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-03-25 16:32:14.309956813 +0000 2024-03-25T18:24:14.967 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-03-25 16:32:14.309956813 +0000 2024-03-25T18:24:14.967 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-03-25 16:32:14.309956813 +0000 2024-03-25T18:24:14.968 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:24:15.034 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-03-25T18:24:15.034 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-03-25T18:24:15.034 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000104462 s, 4.9 MB/s 2024-03-25T18:24:15.035 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:24:15.092 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:24:15.092 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi003.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi016.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:24:15.092 INFO:tasks.ceph:Generating config... 2024-03-25T18:24:15.093 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T18:24:15.094 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T18:24:15.094 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T18:24:15.094 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T18:24:15.094 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T18:24:15.094 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T18:24:15.094 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T18:24:15.094 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T18:24:15.094 INFO:tasks.ceph:[global] ms type = async 2024-03-25T18:24:15.094 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T18:24:15.094 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T18:24:15.094 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T18:24:15.095 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T18:24:15.095 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T18:24:15.095 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T18:24:15.095 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T18:24:15.095 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T18:24:15.095 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T18:24:15.095 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T18:24:15.095 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T18:24:15.095 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T18:24:15.095 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T18:24:15.095 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T18:24:15.095 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T18:24:15.095 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T18:24:15.095 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T18:24:15.095 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T18:24:15.096 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T18:24:15.096 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T18:24:15.096 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T18:24:15.096 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T18:24:15.096 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T18:24:15.096 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T18:24:15.096 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T18:24:15.096 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T18:24:15.096 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T18:24:15.096 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T18:24:15.096 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T18:24:15.096 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T18:24:15.096 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T18:24:15.096 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T18:24:15.096 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T18:24:15.097 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T18:24:15.097 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T18:24:15.097 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T18:24:15.097 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T18:24:15.097 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T18:24:15.097 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T18:24:15.097 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T18:24:15.097 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T18:24:15.097 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T18:24:15.097 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T18:24:15.097 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T18:24:15.097 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T18:24:15.098 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T18:24:15.157 INFO:teuthology.orchestra.run.smithi003.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T18:24:15.159 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T18:24:15.206 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:24:15.233 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.3:6789]'), ('mon.c', '[v1:172.21.15.3:6790]'), ('mon.b', '[v1:172.21.15.16:6789]')] 2024-03-25T18:24:15.233 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.3:6789],[v1:172.21.15.3:6790],[v1:172.21.15.16:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T18:24:15.234 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-25T18:24:15.234 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:24:15.289 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.3:6789]' --addv c '[v1:172.21.15.3:6790]' --addv b '[v1:172.21.15.16:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:15.374 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:15.374 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool: generated fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:24:15.374 INFO:teuthology.orchestra.run.smithi003.stdout:setting min_mon_release = quincy 2024-03-25T18:24:15.374 INFO:teuthology.orchestra.run.smithi003.stdout:epoch 0 2024-03-25T18:24:15.374 INFO:teuthology.orchestra.run.smithi003.stdout:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:24:15.374 INFO:teuthology.orchestra.run.smithi003.stdout:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:24:15.375 INFO:teuthology.orchestra.run.smithi003.stdout:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:24:15.375 INFO:teuthology.orchestra.run.smithi003.stdout:min_mon_release 17 (quincy) 2024-03-25T18:24:15.375 INFO:teuthology.orchestra.run.smithi003.stdout:election_strategy: 3 2024-03-25T18:24:15.375 INFO:teuthology.orchestra.run.smithi003.stdout:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:24:15.375 INFO:teuthology.orchestra.run.smithi003.stdout:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:24:15.375 INFO:teuthology.orchestra.run.smithi003.stdout:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:24:15.375 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T18:24:15.376 DEBUG:teuthology.orchestra.run.smithi003:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:24:15.390 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID da0b8676-a454-4df7-b780-787f16fb595c... 2024-03-25T18:24:15.391 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:24:15.433 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:24:15.480 INFO:teuthology.orchestra.run.smithi003.stdout:[global] 2024-03-25T18:24:15.480 INFO:teuthology.orchestra.run.smithi003.stdout: chdir = "" 2024-03-25T18:24:15.480 INFO:teuthology.orchestra.run.smithi003.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:24:15.480 INFO:teuthology.orchestra.run.smithi003.stdout: auth supported = cephx 2024-03-25T18:24:15.481 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:15.481 INFO:teuthology.orchestra.run.smithi003.stdout: filestore xattr use omap = true 2024-03-25T18:24:15.481 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:15.481 INFO:teuthology.orchestra.run.smithi003.stdout: mon clock drift allowed = 1.000 2024-03-25T18:24:15.481 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:15.481 INFO:teuthology.orchestra.run.smithi003.stdout: osd crush chooseleaf type = 0 2024-03-25T18:24:15.481 INFO:teuthology.orchestra.run.smithi003.stdout: auth debug = true 2024-03-25T18:24:15.481 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:15.481 INFO:teuthology.orchestra.run.smithi003.stdout: ms die on old message = true 2024-03-25T18:24:15.481 INFO:teuthology.orchestra.run.smithi003.stdout: ms die on bug = true 2024-03-25T18:24:15.481 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:15.481 INFO:teuthology.orchestra.run.smithi003.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:24:15.481 INFO:teuthology.orchestra.run.smithi003.stdout: mon pg warn max object skew = 0 2024-03-25T18:24:15.481 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:15.482 INFO:teuthology.orchestra.run.smithi003.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:24:15.482 INFO:teuthology.orchestra.run.smithi003.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:24:15.482 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:15.482 INFO:teuthology.orchestra.run.smithi003.stdout: osd pool default size = 2 2024-03-25T18:24:15.482 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:15.482 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd allow primary affinity = true 2024-03-25T18:24:15.482 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd allow pg remap = true 2024-03-25T18:24:15.482 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:24:15.482 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:24:15.482 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on no sortbitwise = false 2024-03-25T18:24:15.482 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:24:15.482 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on too few osds = false 2024-03-25T18:24:15.482 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:24:15.483 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:24:15.483 INFO:teuthology.orchestra.run.smithi003.stdout: mon_allow_pool_size_one = true 2024-03-25T18:24:15.483 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:15.483 INFO:teuthology.orchestra.run.smithi003.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:24:15.483 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:15.483 INFO:teuthology.orchestra.run.smithi003.stdout: osd default data pool replay window = 5 2024-03-25T18:24:15.483 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:15.483 INFO:teuthology.orchestra.run.smithi003.stdout: mon allow pool delete = true 2024-03-25T18:24:15.483 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:15.483 INFO:teuthology.orchestra.run.smithi003.stdout: mon cluster log file level = debug 2024-03-25T18:24:15.483 INFO:teuthology.orchestra.run.smithi003.stdout: debug asserts on shutdown = true 2024-03-25T18:24:15.483 INFO:teuthology.orchestra.run.smithi003.stdout: mon health detail to clog = false 2024-03-25T18:24:15.483 INFO:teuthology.orchestra.run.smithi003.stdout: mon host = "[v1:172.21.15.3:6789],[v1:172.21.15.3:6790],[v1:172.21.15.16:6789]" 2024-03-25T18:24:15.484 INFO:teuthology.orchestra.run.smithi003.stdout: mon client directed command retry = 5 2024-03-25T18:24:15.484 INFO:teuthology.orchestra.run.smithi003.stdout: mon election default strategy = 3 2024-03-25T18:24:15.484 INFO:teuthology.orchestra.run.smithi003.stdout: ms bind msgr2 = False 2024-03-25T18:24:15.484 INFO:teuthology.orchestra.run.smithi003.stdout: ms inject socket failures = 5000 2024-03-25T18:24:15.484 INFO:teuthology.orchestra.run.smithi003.stdout: ms type = async 2024-03-25T18:24:15.484 INFO:teuthology.orchestra.run.smithi003.stdout: osd heartbeat grace = 80 2024-03-25T18:24:15.484 INFO:teuthology.orchestra.run.smithi003.stdout: fsid = da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:24:15.484 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:15.484 INFO:teuthology.orchestra.run.smithi003.stdout:[osd] 2024-03-25T18:24:15.484 INFO:teuthology.orchestra.run.smithi003.stdout: osd journal size = 100 2024-03-25T18:24:15.484 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:15.484 INFO:teuthology.orchestra.run.smithi003.stdout: osd scrub load threshold = 5.0 2024-03-25T18:24:15.484 INFO:teuthology.orchestra.run.smithi003.stdout: osd scrub max interval = 600 2024-03-25T18:24:15.484 INFO:teuthology.orchestra.run.smithi003.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:24:15.484 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:15.485 INFO:teuthology.orchestra.run.smithi003.stdout: osd recover clone overlap = true 2024-03-25T18:24:15.485 INFO:teuthology.orchestra.run.smithi003.stdout: osd recovery max chunk = 1048576 2024-03-25T18:24:15.485 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:15.485 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug shutdown = true 2024-03-25T18:24:15.485 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug op order = true 2024-03-25T18:24:15.485 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug verify stray on activate = true 2024-03-25T18:24:15.485 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug trim objects = true 2024-03-25T18:24:15.485 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:15.485 INFO:teuthology.orchestra.run.smithi003.stdout: osd open classes on start = true 2024-03-25T18:24:15.485 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug pg log writeout = true 2024-03-25T18:24:15.485 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:15.485 INFO:teuthology.orchestra.run.smithi003.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:24:15.485 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:15.485 INFO:teuthology.orchestra.run.smithi003.stdout: osd map max advance = 10 2024-03-25T18:24:15.486 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:15.486 INFO:teuthology.orchestra.run.smithi003.stdout: journal zero on create = true 2024-03-25T18:24:15.486 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:15.486 INFO:teuthology.orchestra.run.smithi003.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:24:15.486 INFO:teuthology.orchestra.run.smithi003.stdout: filestore apply finisher threads = 3 2024-03-25T18:24:15.486 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:15.486 INFO:teuthology.orchestra.run.smithi003.stdout: bdev debug aio = true 2024-03-25T18:24:15.486 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug misdirected ops = true 2024-03-25T18:24:15.486 INFO:teuthology.orchestra.run.smithi003.stdout: bdev async discard = True 2024-03-25T18:24:15.486 INFO:teuthology.orchestra.run.smithi003.stdout: bdev enable discard = True 2024-03-25T18:24:15.486 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore allocator = stupid 2024-03-25T18:24:15.486 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore block size = 96636764160 2024-03-25T18:24:15.486 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore fsck on mount = True 2024-03-25T18:24:15.487 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore zero block detection = True 2024-03-25T18:24:15.487 INFO:teuthology.orchestra.run.smithi003.stdout: debug bluefs = 1 2024-03-25T18:24:15.487 INFO:teuthology.orchestra.run.smithi003.stdout: debug bluestore = 1 2024-03-25T18:24:15.487 INFO:teuthology.orchestra.run.smithi003.stdout: debug monc = 20 2024-03-25T18:24:15.487 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-03-25T18:24:15.487 INFO:teuthology.orchestra.run.smithi003.stdout: debug osd = 20 2024-03-25T18:24:15.487 INFO:teuthology.orchestra.run.smithi003.stdout: debug rocksdb = 10 2024-03-25T18:24:15.487 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:24:15.487 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd full ratio = 0.9 2024-03-25T18:24:15.487 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:24:15.487 INFO:teuthology.orchestra.run.smithi003.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:24:15.487 INFO:teuthology.orchestra.run.smithi003.stdout: osd class default list = * 2024-03-25T18:24:15.487 INFO:teuthology.orchestra.run.smithi003.stdout: osd class load list = * 2024-03-25T18:24:15.487 INFO:teuthology.orchestra.run.smithi003.stdout: osd client watch timeout = 120 2024-03-25T18:24:15.487 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug verify cached snaps = True 2024-03-25T18:24:15.488 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug verify missing on start = True 2024-03-25T18:24:15.488 INFO:teuthology.orchestra.run.smithi003.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:24:15.488 INFO:teuthology.orchestra.run.smithi003.stdout: osd fast shutdown = False 2024-03-25T18:24:15.488 INFO:teuthology.orchestra.run.smithi003.stdout: osd max markdown count = 1000 2024-03-25T18:24:15.488 INFO:teuthology.orchestra.run.smithi003.stdout: osd mclock override recovery settings = True 2024-03-25T18:24:15.488 INFO:teuthology.orchestra.run.smithi003.stdout: osd objectstore = bluestore 2024-03-25T18:24:15.488 INFO:teuthology.orchestra.run.smithi003.stdout: osd op queue = debug_random 2024-03-25T18:24:15.488 INFO:teuthology.orchestra.run.smithi003.stdout: osd op queue cut off = debug_random 2024-03-25T18:24:15.488 INFO:teuthology.orchestra.run.smithi003.stdout: osd shutdown pgref assert = True 2024-03-25T18:24:15.488 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:15.488 INFO:teuthology.orchestra.run.smithi003.stdout:[mgr] 2024-03-25T18:24:15.488 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-03-25T18:24:15.488 INFO:teuthology.orchestra.run.smithi003.stdout: debug mgr = 20 2024-03-25T18:24:15.488 INFO:teuthology.orchestra.run.smithi003.stdout: debug mon = 20 2024-03-25T18:24:15.488 INFO:teuthology.orchestra.run.smithi003.stdout: debug auth = 20 2024-03-25T18:24:15.488 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:24:15.489 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:24:15.489 INFO:teuthology.orchestra.run.smithi003.stdout: mgr/telemetry/nag = false 2024-03-25T18:24:15.489 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:15.489 INFO:teuthology.orchestra.run.smithi003.stdout:[mon] 2024-03-25T18:24:15.489 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-03-25T18:24:15.489 INFO:teuthology.orchestra.run.smithi003.stdout: debug mon = 20 2024-03-25T18:24:15.489 INFO:teuthology.orchestra.run.smithi003.stdout: debug paxos = 20 2024-03-25T18:24:15.489 INFO:teuthology.orchestra.run.smithi003.stdout: debug auth = 20 2024-03-25T18:24:15.489 INFO:teuthology.orchestra.run.smithi003.stdout: mon data avail warn = 5 2024-03-25T18:24:15.489 INFO:teuthology.orchestra.run.smithi003.stdout: mon mgr mkfs grace = 240 2024-03-25T18:24:15.489 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:24:15.489 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd reporter subtree level = osd 2024-03-25T18:24:15.489 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd prime pg temp = true 2024-03-25T18:24:15.490 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:24:15.490 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:15.490 INFO:teuthology.orchestra.run.smithi003.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:24:15.490 INFO:teuthology.orchestra.run.smithi003.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:24:15.490 INFO:teuthology.orchestra.run.smithi003.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:24:15.490 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:15.490 INFO:teuthology.orchestra.run.smithi003.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:24:15.490 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:24:15.490 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:24:15.490 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:15.490 INFO:teuthology.orchestra.run.smithi003.stdout: # 1m isn't quite enough 2024-03-25T18:24:15.490 INFO:teuthology.orchestra.run.smithi003.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:24:15.490 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:15.491 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:24:15.491 INFO:teuthology.orchestra.run.smithi003.stdout: mon min osdmap epochs = 50 2024-03-25T18:24:15.491 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd crush smoke test = False 2024-03-25T18:24:15.491 INFO:teuthology.orchestra.run.smithi003.stdout: mon osdmap full prune interval = 2 2024-03-25T18:24:15.491 INFO:teuthology.orchestra.run.smithi003.stdout: mon osdmap full prune min = 15 2024-03-25T18:24:15.491 INFO:teuthology.orchestra.run.smithi003.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:24:15.491 INFO:teuthology.orchestra.run.smithi003.stdout: mon scrub interval = 300 2024-03-25T18:24:15.491 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on pool no app = False 2024-03-25T18:24:15.491 INFO:teuthology.orchestra.run.smithi003.stdout: paxos service trim min = 10 2024-03-25T18:24:15.491 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:24:15.491 INFO:teuthology.orchestra.run.smithi003.stdout:[client] 2024-03-25T18:24:15.491 INFO:teuthology.orchestra.run.smithi003.stdout: rgw cache enabled = true 2024-03-25T18:24:15.491 INFO:teuthology.orchestra.run.smithi003.stdout: rgw enable ops log = true 2024-03-25T18:24:15.491 INFO:teuthology.orchestra.run.smithi003.stdout: rgw enable usage log = true 2024-03-25T18:24:15.491 INFO:teuthology.orchestra.run.smithi003.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:24:15.492 INFO:teuthology.orchestra.run.smithi003.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:24:15.492 INFO:teuthology.orchestra.run.smithi003.stdout: debug monc = 20 2024-03-25T18:24:15.492 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-03-25T18:24:15.492 INFO:teuthology.orchestra.run.smithi003.stdout: debug objecter = 20 2024-03-25T18:24:15.492 INFO:teuthology.orchestra.run.smithi003.stdout: debug rados = 20 2024-03-25T18:24:15.492 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.a] 2024-03-25T18:24:15.492 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.c] 2024-03-25T18:24:15.492 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.b] 2024-03-25T18:24:15.492 INFO:teuthology.orchestra.run.smithi016.stdout:[global] 2024-03-25T18:24:15.492 INFO:teuthology.orchestra.run.smithi016.stdout: chdir = "" 2024-03-25T18:24:15.492 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:24:15.493 INFO:teuthology.orchestra.run.smithi016.stdout: auth supported = cephx 2024-03-25T18:24:15.493 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:24:15.493 INFO:teuthology.orchestra.run.smithi016.stdout: filestore xattr use omap = true 2024-03-25T18:24:15.493 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:24:15.493 INFO:teuthology.orchestra.run.smithi016.stdout: mon clock drift allowed = 1.000 2024-03-25T18:24:15.493 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:24:15.493 INFO:teuthology.orchestra.run.smithi016.stdout: osd crush chooseleaf type = 0 2024-03-25T18:24:15.493 INFO:teuthology.orchestra.run.smithi016.stdout: auth debug = true 2024-03-25T18:24:15.493 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:24:15.493 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on old message = true 2024-03-25T18:24:15.493 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on bug = true 2024-03-25T18:24:15.493 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:24:15.493 INFO:teuthology.orchestra.run.smithi016.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:24:15.493 INFO:teuthology.orchestra.run.smithi016.stdout: mon pg warn max object skew = 0 2024-03-25T18:24:15.494 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:24:15.494 INFO:teuthology.orchestra.run.smithi016.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:24:15.494 INFO:teuthology.orchestra.run.smithi016.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:24:15.494 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:24:15.494 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default size = 2 2024-03-25T18:24:15.494 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:24:15.494 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow primary affinity = true 2024-03-25T18:24:15.494 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow pg remap = true 2024-03-25T18:24:15.494 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:24:15.494 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:24:15.494 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on no sortbitwise = false 2024-03-25T18:24:15.494 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:24:15.495 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on too few osds = false 2024-03-25T18:24:15.495 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:24:15.495 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:24:15.495 INFO:teuthology.orchestra.run.smithi016.stdout: mon_allow_pool_size_one = true 2024-03-25T18:24:15.495 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:24:15.495 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:24:15.495 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:24:15.495 INFO:teuthology.orchestra.run.smithi016.stdout: osd default data pool replay window = 5 2024-03-25T18:24:15.495 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:24:15.495 INFO:teuthology.orchestra.run.smithi016.stdout: mon allow pool delete = true 2024-03-25T18:24:15.495 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:24:15.495 INFO:teuthology.orchestra.run.smithi016.stdout: mon cluster log file level = debug 2024-03-25T18:24:15.495 INFO:teuthology.orchestra.run.smithi016.stdout: debug asserts on shutdown = true 2024-03-25T18:24:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: mon health detail to clog = false 2024-03-25T18:24:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: mon host = "[v1:172.21.15.3:6789],[v1:172.21.15.3:6790],[v1:172.21.15.16:6789]" 2024-03-25T18:24:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: mon client directed command retry = 5 2024-03-25T18:24:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: mon election default strategy = 3 2024-03-25T18:24:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: ms bind msgr2 = False 2024-03-25T18:24:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: ms inject socket failures = 5000 2024-03-25T18:24:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: ms type = async 2024-03-25T18:24:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: osd heartbeat grace = 80 2024-03-25T18:24:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: fsid = da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:24:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:24:15.496 INFO:teuthology.orchestra.run.smithi016.stdout:[osd] 2024-03-25T18:24:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: osd journal size = 100 2024-03-25T18:24:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:24:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub load threshold = 5.0 2024-03-25T18:24:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub max interval = 600 2024-03-25T18:24:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:24:15.497 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:24:15.497 INFO:teuthology.orchestra.run.smithi016.stdout: osd recover clone overlap = true 2024-03-25T18:24:15.497 INFO:teuthology.orchestra.run.smithi016.stdout: osd recovery max chunk = 1048576 2024-03-25T18:24:15.497 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:24:15.497 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug shutdown = true 2024-03-25T18:24:15.497 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug op order = true 2024-03-25T18:24:15.497 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug verify stray on activate = true 2024-03-25T18:24:15.497 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug trim objects = true 2024-03-25T18:24:15.497 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:24:15.497 INFO:teuthology.orchestra.run.smithi016.stdout: osd open classes on start = true 2024-03-25T18:24:15.497 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug pg log writeout = true 2024-03-25T18:24:15.497 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:24:15.497 INFO:teuthology.orchestra.run.smithi016.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:24:15.497 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:24:15.498 INFO:teuthology.orchestra.run.smithi016.stdout: osd map max advance = 10 2024-03-25T18:24:15.498 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:24:15.498 INFO:teuthology.orchestra.run.smithi016.stdout: journal zero on create = true 2024-03-25T18:24:15.498 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:24:15.498 INFO:teuthology.orchestra.run.smithi016.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:24:15.498 INFO:teuthology.orchestra.run.smithi016.stdout: filestore apply finisher threads = 3 2024-03-25T18:24:15.498 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:24:15.498 INFO:teuthology.orchestra.run.smithi016.stdout: bdev debug aio = true 2024-03-25T18:24:15.498 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug misdirected ops = true 2024-03-25T18:24:15.498 INFO:teuthology.orchestra.run.smithi016.stdout: bdev async discard = True 2024-03-25T18:24:15.498 INFO:teuthology.orchestra.run.smithi016.stdout: bdev enable discard = True 2024-03-25T18:24:15.498 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore allocator = stupid 2024-03-25T18:24:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore block size = 96636764160 2024-03-25T18:24:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore fsck on mount = True 2024-03-25T18:24:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore zero block detection = True 2024-03-25T18:24:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluefs = 1 2024-03-25T18:24:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluestore = 1 2024-03-25T18:24:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: debug monc = 20 2024-03-25T18:24:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-03-25T18:24:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: debug osd = 20 2024-03-25T18:24:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: debug rocksdb = 10 2024-03-25T18:24:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:24:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd full ratio = 0.9 2024-03-25T18:24:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:24:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:24:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: osd class default list = * 2024-03-25T18:24:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: osd class load list = * 2024-03-25T18:24:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: osd client watch timeout = 120 2024-03-25T18:24:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug verify cached snaps = True 2024-03-25T18:24:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug verify missing on start = True 2024-03-25T18:24:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:24:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: osd fast shutdown = False 2024-03-25T18:24:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: osd max markdown count = 1000 2024-03-25T18:24:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: osd mclock override recovery settings = True 2024-03-25T18:24:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: osd objectstore = bluestore 2024-03-25T18:24:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: osd op queue = debug_random 2024-03-25T18:24:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: osd op queue cut off = debug_random 2024-03-25T18:24:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: osd shutdown pgref assert = True 2024-03-25T18:24:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:24:15.500 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr] 2024-03-25T18:24:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-03-25T18:24:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: debug mgr = 20 2024-03-25T18:24:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2024-03-25T18:24:15.501 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2024-03-25T18:24:15.501 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:24:15.501 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:24:15.501 INFO:teuthology.orchestra.run.smithi016.stdout: mgr/telemetry/nag = false 2024-03-25T18:24:15.501 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:24:15.501 INFO:teuthology.orchestra.run.smithi016.stdout:[mon] 2024-03-25T18:24:15.501 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-03-25T18:24:15.501 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2024-03-25T18:24:15.501 INFO:teuthology.orchestra.run.smithi016.stdout: debug paxos = 20 2024-03-25T18:24:15.501 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2024-03-25T18:24:15.501 INFO:teuthology.orchestra.run.smithi016.stdout: mon data avail warn = 5 2024-03-25T18:24:15.501 INFO:teuthology.orchestra.run.smithi016.stdout: mon mgr mkfs grace = 240 2024-03-25T18:24:15.501 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:24:15.501 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd reporter subtree level = osd 2024-03-25T18:24:15.501 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd prime pg temp = true 2024-03-25T18:24:15.501 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:24:15.502 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:24:15.502 INFO:teuthology.orchestra.run.smithi016.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:24:15.502 INFO:teuthology.orchestra.run.smithi016.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:24:15.502 INFO:teuthology.orchestra.run.smithi016.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:24:15.502 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:24:15.502 INFO:teuthology.orchestra.run.smithi016.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:24:15.502 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:24:15.502 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:24:15.502 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:24:15.502 INFO:teuthology.orchestra.run.smithi016.stdout: # 1m isn't quite enough 2024-03-25T18:24:15.502 INFO:teuthology.orchestra.run.smithi016.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:24:15.502 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:24:15.502 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:24:15.502 INFO:teuthology.orchestra.run.smithi016.stdout: mon min osdmap epochs = 50 2024-03-25T18:24:15.503 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd crush smoke test = False 2024-03-25T18:24:15.503 INFO:teuthology.orchestra.run.smithi016.stdout: mon osdmap full prune interval = 2 2024-03-25T18:24:15.503 INFO:teuthology.orchestra.run.smithi016.stdout: mon osdmap full prune min = 15 2024-03-25T18:24:15.503 INFO:teuthology.orchestra.run.smithi016.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:24:15.503 INFO:teuthology.orchestra.run.smithi016.stdout: mon scrub interval = 300 2024-03-25T18:24:15.503 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on pool no app = False 2024-03-25T18:24:15.503 INFO:teuthology.orchestra.run.smithi016.stdout: paxos service trim min = 10 2024-03-25T18:24:15.503 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-25T18:24:15.503 INFO:teuthology.orchestra.run.smithi016.stdout:[client] 2024-03-25T18:24:15.503 INFO:teuthology.orchestra.run.smithi016.stdout: rgw cache enabled = true 2024-03-25T18:24:15.503 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable ops log = true 2024-03-25T18:24:15.503 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable usage log = true 2024-03-25T18:24:15.503 INFO:teuthology.orchestra.run.smithi016.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:24:15.503 INFO:teuthology.orchestra.run.smithi016.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:24:15.503 INFO:teuthology.orchestra.run.smithi016.stdout: debug monc = 20 2024-03-25T18:24:15.503 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-03-25T18:24:15.504 INFO:teuthology.orchestra.run.smithi016.stdout: debug objecter = 20 2024-03-25T18:24:15.504 INFO:teuthology.orchestra.run.smithi016.stdout: debug rados = 20 2024-03-25T18:24:15.504 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.a] 2024-03-25T18:24:15.504 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.c] 2024-03-25T18:24:15.504 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.b] 2024-03-25T18:24:15.505 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T18:24:15.505 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T18:24:15.560 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T18:24:15.560 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-25T18:24:15.560 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T18:24:15.574 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-25T18:24:15.574 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T18:24:15.629 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi003.front.sepia.ceph.com 2024-03-25T18:24:15.629 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-25T18:24:15.629 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:24:15.629 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:24:15.707 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-25T18:24:15.707 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:15.762 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi016.front.sepia.ceph.com 2024-03-25T18:24:15.762 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-25T18:24:15.762 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:24:15.762 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:24:15.807 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-25T18:24:15.807 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:15.864 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T18:24:15.864 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T18:24:15.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:24:15.918 INFO:teuthology.orchestra.run.smithi003.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:24:15.921 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:24:16.008 INFO:teuthology.orchestra.run.smithi016.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:24:16.011 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T18:24:16.011 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T18:24:16.011 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T18:24:16.052 INFO:teuthology.orchestra.run.smithi003.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T18:24:16.068 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T18:24:16.112 INFO:teuthology.orchestra.run.smithi016.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T18:24:16.128 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T18:24:16.128 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi003.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi016.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:24:16.128 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T18:24:16.155 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:24:16.155 INFO:tasks.ceph:role: osd.0 2024-03-25T18:24:16.155 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi003.front.sepia.ceph.com 2024-03-25T18:24:16.155 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:24:16.228 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:24:16.229 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:24:16.229 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:24:16.229 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:24:16.229 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:24:16.229 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:24:16.229 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:24:16.229 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:24:16.229 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:24:16.229 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:24:16.745 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2024-03-25T18:24:16.746 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2024-03-25T18:24:16.746 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T18:24:16.871 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T18:24:16.903 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T18:24:16.970 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:24:16.970 INFO:tasks.ceph:role: osd.1 2024-03-25T18:24:16.970 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi003.front.sepia.ceph.com 2024-03-25T18:24:16.970 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:24:17.044 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:24:17.044 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:24:17.044 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:24:17.044 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:24:17.044 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:24:17.044 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:24:17.045 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:24:17.045 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:24:17.045 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:24:17.045 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:24:17.486 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2024-03-25T18:24:17.488 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2024-03-25T18:24:17.489 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T18:24:17.573 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T18:24:17.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T18:24:17.711 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:24:17.711 INFO:tasks.ceph:role: osd.2 2024-03-25T18:24:17.711 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi003.front.sepia.ceph.com 2024-03-25T18:24:17.711 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:24:17.784 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:24:17.784 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:24:17.784 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:24:17.784 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:24:17.784 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:24:17.784 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:24:17.784 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:24:17.784 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:24:17.784 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:24:17.784 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:24:18.299 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2024-03-25T18:24:18.300 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2024-03-25T18:24:18.301 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T18:24:18.427 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T18:24:18.457 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T18:24:18.523 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:24:18.523 INFO:tasks.ceph:role: osd.3 2024-03-25T18:24:18.523 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi003.front.sepia.ceph.com 2024-03-25T18:24:18.523 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:24:18.597 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:24:18.597 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:24:18.597 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:24:18.597 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:24:18.597 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:24:18.597 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:24:18.597 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:24:18.597 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:24:18.597 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:24:18.597 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:24:19.116 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2024-03-25T18:24:19.119 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2024-03-25T18:24:19.119 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T18:24:19.237 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T18:24:19.271 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:19.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:19.363+0000 7f0997217640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T18:24:19.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:19.363+0000 7f0997217640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T18:24:19.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:19.363+0000 7f0997217640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:24:19.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:19.363+0000 7f0997217640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:24:19.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:19.363+0000 7f0997217640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:24:19.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:19.364+0000 7f0997217640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T18:24:22.213 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T18:24:22.244 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:22.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:22.333+0000 7f56cee9d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T18:24:22.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:22.333+0000 7f56cee9d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T18:24:22.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:22.333+0000 7f56cee9d640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:24:22.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:22.333+0000 7f56cee9d640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:24:22.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:22.333+0000 7f56cee9d640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:24:22.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:22.334+0000 7f56cee9d640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T18:24:25.176 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T18:24:25.207 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:25.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:25.296+0000 7f8301e15640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T18:24:25.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:25.297+0000 7f8301e15640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T18:24:25.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:25.297+0000 7f8301e15640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:24:25.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:25.297+0000 7f8301e15640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:24:25.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:25.297+0000 7f8301e15640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:24:25.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:25.297+0000 7f8301e15640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T18:24:28.149 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T18:24:28.179 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:28.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:28.269+0000 7f1040611640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T18:24:28.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:28.269+0000 7f1040611640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T18:24:28.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:28.269+0000 7f1040611640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:24:28.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:28.269+0000 7f1040611640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:24:28.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:28.269+0000 7f1040611640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:24:28.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:28.270+0000 7f1040611640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T18:24:31.115 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T18:24:31.146 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T18:24:31.178 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:24:31.178 INFO:tasks.ceph:role: osd.4 2024-03-25T18:24:31.178 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi016.front.sepia.ceph.com 2024-03-25T18:24:31.178 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:24:31.248 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:24:31.248 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:24:31.248 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:24:31.248 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:24:31.248 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:24:31.248 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:24:31.248 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:24:31.248 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:24:31.248 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:24:31.248 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:24:31.280 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2024-03-25T18:24:31.282 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2024-03-25T18:24:31.282 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T18:24:31.355 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T18:24:31.426 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T18:24:31.494 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:24:31.494 INFO:tasks.ceph:role: osd.5 2024-03-25T18:24:31.494 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi016.front.sepia.ceph.com 2024-03-25T18:24:31.494 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:24:31.560 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:24:31.560 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:24:31.560 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:24:31.561 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:24:31.561 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:24:31.561 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:24:31.561 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:24:31.561 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:24:31.561 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:24:31.561 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:24:31.594 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2024-03-25T18:24:31.596 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2024-03-25T18:24:31.596 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T18:24:31.666 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T18:24:31.737 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T18:24:31.803 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:24:31.803 INFO:tasks.ceph:role: osd.6 2024-03-25T18:24:31.803 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi016.front.sepia.ceph.com 2024-03-25T18:24:31.803 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:24:31.868 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:24:31.869 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:24:31.869 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:24:31.869 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:24:31.869 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:24:31.869 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:24:31.869 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:24:31.869 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:24:31.869 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:24:31.869 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:24:31.901 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2024-03-25T18:24:31.903 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2024-03-25T18:24:31.903 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T18:24:31.972 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T18:24:32.042 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T18:24:32.109 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:24:32.109 INFO:tasks.ceph:role: osd.7 2024-03-25T18:24:32.109 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi016.front.sepia.ceph.com 2024-03-25T18:24:32.110 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:24:32.175 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:24:32.175 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:24:32.175 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:24:32.175 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:24:32.175 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:24:32.175 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:24:32.175 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:24:32.176 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:24:32.176 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:24:32.176 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:24:32.208 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2024-03-25T18:24:32.210 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2024-03-25T18:24:32.210 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T18:24:32.280 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T18:24:32.348 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:32.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:24:32.440+0000 7f31c5c0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T18:24:32.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:24:32.440+0000 7f31c5c0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T18:24:32.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:24:32.440+0000 7f31c5c0d640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:24:32.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:24:32.440+0000 7f31c5c0d640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:24:32.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:24:32.440+0000 7f31c5c0d640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:24:32.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:24:32.441+0000 7f31c5c0d640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T18:24:35.276 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T18:24:35.307 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:35.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:24:35.398+0000 7f11b3bfd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T18:24:35.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:24:35.398+0000 7f11b3bfd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T18:24:35.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:24:35.398+0000 7f11b3bfd640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:24:35.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:24:35.398+0000 7f11b3bfd640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:24:35.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:24:35.398+0000 7f11b3bfd640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:24:35.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:24:35.399+0000 7f11b3bfd640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T18:24:38.235 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T18:24:38.266 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:38.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:24:38.358+0000 7fa8d05e1640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T18:24:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:24:38.358+0000 7fa8d05e1640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T18:24:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:24:38.358+0000 7fa8d05e1640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:24:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:24:38.358+0000 7fa8d05e1640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:24:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:24:38.358+0000 7fa8d05e1640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:24:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:24:38.359+0000 7fa8d05e1640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T18:24:41.204 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T18:24:41.235 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:41.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:24:41.325+0000 7f3d4f6c4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T18:24:41.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:24:41.325+0000 7f3d4f6c4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T18:24:41.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:24:41.325+0000 7f3d4f6c4640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:24:41.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:24:41.325+0000 7f3d4f6c4640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:24:41.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:24:41.325+0000 7f3d4f6c4640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:24:41.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:24:41.325+0000 7f3d4f6c4640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T18:24:44.164 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T18:24:44.195 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T18:24:44.195 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-25T18:24:44.195 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T18:24:44.227 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-25T18:24:44.227 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T18:24:44.297 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-25T18:24:44.298 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T18:24:44.367 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-25T18:24:44.368 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T18:24:44.436 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-25T18:24:44.437 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T18:24:44.505 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-25T18:24:44.506 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T18:24:44.536 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-25T18:24:44.536 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T18:24:44.607 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-25T18:24:44.607 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T18:24:44.675 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-25T18:24:44.676 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T18:24:44.743 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-25T18:24:44.744 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T18:24:44.811 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-25T18:24:44.811 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T18:24:44.828 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-25T18:24:44.828 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T18:24:44.866 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T18:24:44.866 DEBUG:teuthology.orchestra.run.smithi003:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:24:44.871 DEBUG:teuthology.orchestra.run.smithi016:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:24:44.908 INFO:teuthology.orchestra.run.smithi003.stdout:[mgr.y] 2024-03-25T18:24:44.908 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBPwQFmWyPDNhAAWxXfhQMh3yfTqrR8HtJYww== 2024-03-25T18:24:44.908 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.0] 2024-03-25T18:24:44.908 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBTwQFmWzO4FRAAb1esEMJrmd/6rXQmpE6pew== 2024-03-25T18:24:44.909 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.1] 2024-03-25T18:24:44.909 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBWwQFmgzDvExAAMFYpCu5lF89Br1y52uUQoQ== 2024-03-25T18:24:44.909 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.2] 2024-03-25T18:24:44.909 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBZwQFmQZfDERAAc0RnMBi3iwvZg+C3/lHRYQ== 2024-03-25T18:24:44.909 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.3] 2024-03-25T18:24:44.909 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBcwQFmm2siEBAA/INqRlZ+tPaGei7Ppuxtig== 2024-03-25T18:24:44.909 INFO:teuthology.orchestra.run.smithi003.stdout:[mgr.x] 2024-03-25T18:24:44.909 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBQwQFmG4SGABAAj9jGO2D4vhgFq3uqeHPk2w== 2024-03-25T18:24:44.909 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.4] 2024-03-25T18:24:44.909 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBgwQFmK9ZTGhAADbkiL5veGw5LwxF3NVA4IA== 2024-03-25T18:24:44.909 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.5] 2024-03-25T18:24:44.909 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBjwQFmojfRFxAAc3n2Pd3KF01vrHyFdZjfzg== 2024-03-25T18:24:44.909 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.6] 2024-03-25T18:24:44.909 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBmwQFm9Y5uFRAADtHMDlW+uMlr2EGOjpVNrg== 2024-03-25T18:24:44.910 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.7] 2024-03-25T18:24:44.910 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBpwQFmkON0ExAACsPQvxIdB/8SNFrklpg6LA== 2024-03-25T18:24:44.910 INFO:teuthology.orchestra.run.smithi003.stdout:[client.0] 2024-03-25T18:24:44.910 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBQwQFm95AqAxAAS2l6EZxdgmLe3OOQkgCdzw== 2024-03-25T18:24:44.910 INFO:teuthology.orchestra.run.smithi003.stdout:[client.1] 2024-03-25T18:24:44.910 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBQwQFm7xK5BhAA5ih93vR/6aNKFKu6wICp1A== 2024-03-25T18:24:44.931 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr.y] 2024-03-25T18:24:44.931 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQBPwQFmWyPDNhAAWxXfhQMh3yfTqrR8HtJYww== 2024-03-25T18:24:44.931 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.0] 2024-03-25T18:24:44.931 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQBTwQFmWzO4FRAAb1esEMJrmd/6rXQmpE6pew== 2024-03-25T18:24:44.931 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.1] 2024-03-25T18:24:44.931 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQBWwQFmgzDvExAAMFYpCu5lF89Br1y52uUQoQ== 2024-03-25T18:24:44.931 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.2] 2024-03-25T18:24:44.932 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQBZwQFmQZfDERAAc0RnMBi3iwvZg+C3/lHRYQ== 2024-03-25T18:24:44.932 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.3] 2024-03-25T18:24:44.932 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQBcwQFmm2siEBAA/INqRlZ+tPaGei7Ppuxtig== 2024-03-25T18:24:44.932 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr.x] 2024-03-25T18:24:44.932 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQBQwQFmG4SGABAAj9jGO2D4vhgFq3uqeHPk2w== 2024-03-25T18:24:44.932 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.4] 2024-03-25T18:24:44.932 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQBgwQFmK9ZTGhAADbkiL5veGw5LwxF3NVA4IA== 2024-03-25T18:24:44.932 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.5] 2024-03-25T18:24:44.932 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQBjwQFmojfRFxAAc3n2Pd3KF01vrHyFdZjfzg== 2024-03-25T18:24:44.932 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.6] 2024-03-25T18:24:44.932 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQBmwQFm9Y5uFRAADtHMDlW+uMlr2EGOjpVNrg== 2024-03-25T18:24:44.932 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.7] 2024-03-25T18:24:44.932 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQBpwQFmkON0ExAACsPQvxIdB/8SNFrklpg6LA== 2024-03-25T18:24:44.932 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2024-03-25T18:24:44.932 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQBQwQFm95AqAxAAS2l6EZxdgmLe3OOQkgCdzw== 2024-03-25T18:24:44.932 INFO:teuthology.orchestra.run.smithi016.stdout:[client.1] 2024-03-25T18:24:44.933 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQBQwQFm7xK5BhAA5ih93vR/6aNKFKu6wICp1A== 2024-03-25T18:24:44.933 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:24:44.953 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:24:45.026 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:45.028 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:45.079 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:45.081 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:45.132 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:45.134 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:45.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:45.186 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:45.237 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:24:45.239 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:24:45.290 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:45.292 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:45.343 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:45.345 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:45.395 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:45.398 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:45.449 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:45.451 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:45.501 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:24:45.503 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:24:45.557 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:24:45.559 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:24:45.613 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T18:24:45.613 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T18:24:45.639 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:24:45.734 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T18:24:45.760 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T18:24:45.827 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:24:45.923 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T18:24:45.950 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T18:24:45.977 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:24:46.074 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T18:24:46.100 DEBUG:teuthology.orchestra.run.smithi003:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:46.102 DEBUG:teuthology.orchestra.run.smithi016:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:46.155 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T18:24:46.155 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:24:46.155 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T18:24:46.155 DEBUG:teuthology.orchestra.run.smithi003:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T18:24:46.157 INFO:tasks.ceph.mon.a:Started 2024-03-25T18:24:46.158 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:24:46.158 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T18:24:46.158 DEBUG:teuthology.orchestra.run.smithi003:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T18:24:46.160 INFO:tasks.ceph.mon.c:Started 2024-03-25T18:24:46.160 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:24:46.160 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T18:24:46.160 DEBUG:teuthology.orchestra.run.smithi016:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T18:24:46.197 INFO:tasks.ceph.mon.b:Started 2024-03-25T18:24:46.197 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T18:24:46.197 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T18:24:46.198 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T18:24:46.200 INFO:tasks.ceph.mgr.y:Started 2024-03-25T18:24:46.200 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T18:24:46.201 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T18:24:46.202 INFO:tasks.ceph.mgr.x:Started 2024-03-25T18:24:46.202 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T18:24:46.203 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T18:24:46.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:46.304+0000 7f42242ba640 10 monclient: get_monmap_and_config 2024-03-25T18:24:46.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:46.304+0000 7f42242ba640 10 monclient: build_initial_monmap 2024-03-25T18:24:46.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:46.304+0000 7f42242ba640 10 monclient: monmap: 2024-03-25T18:24:46.304 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-03-25T18:24:46.304 INFO:teuthology.orchestra.run.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:24:46.304 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-03-25T18:24:46.304169+0000 2024-03-25T18:24:46.304 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-03-25T18:24:46.304169+0000 2024-03-25T18:24:46.304 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:46.304 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-03-25T18:24:46.304 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:24:46.304 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:24:46.304 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:24:46.304 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-25T18:24:46.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:46.304+0000 7f42242ba640 1 Processor -- start 2024-03-25T18:24:46.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:46.304+0000 7f42242ba640 1 -- start start 2024-03-25T18:24:46.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:46.304+0000 7f42242ba640 10 monclient: init 2024-03-25T18:24:46.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:46.305+0000 7f42242ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:46.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:46.305+0000 7f42242ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:46.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:46.305+0000 7f42242ba640 10 monclient(hunting): picked mon.noname-c con 0x7f421c0b4f00 addr v1:172.21.15.16:6789/0 2024-03-25T18:24:46.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:46.305+0000 7f42242ba640 10 monclient(hunting): picked mon.noname-b con 0x7f421c0b4510 addr v1:172.21.15.3:6790/0 2024-03-25T18:24:46.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:46.305+0000 7f42242ba640 10 monclient(hunting): picked mon.noname-a con 0x7f421c142ad0 addr v1:172.21.15.3:6789/0 2024-03-25T18:24:46.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:46.305+0000 7f42242ba640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f421c0a9d90 con 0x7f421c142ad0 2024-03-25T18:24:46.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:46.305+0000 7f42242ba640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f421c0aa2b0 con 0x7f421c0b4f00 2024-03-25T18:24:46.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:46.305+0000 7f42242ba640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f421c13fc60 con 0x7f421c0b4510 2024-03-25T18:24:46.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:46.305+0000 7f42242ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:46.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:46.305+0000 7f42242ba640 10 monclient(hunting): authenticate will time out at 7340.564799s 2024-03-25T18:24:46.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:46.305+0000 7f4222857640 1 -- >> v1:172.21.15.3:6789/0 conn(0x7f421c142ad0 legacy=0x7f421c14dc80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6789/0 2024-03-25T18:24:46.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:46.305+0000 7f4221855640 1 -- >> v1:172.21.15.3:6790/0 conn(0x7f421c0b4510 legacy=0x7f421c13f550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6790/0 2024-03-25T18:24:46.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:46.305+0000 7f4222056640 1 -- >> v1:172.21.15.16:6789/0 conn(0x7f421c0b4f00 legacy=0x7f421c1423c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.16:6789/0 2024-03-25T18:24:46.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:46.506+0000 7f4222857640 1 -- >> v1:172.21.15.3:6789/0 conn(0x7f421c142ad0 legacy=0x7f421c14dc80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6789/0 2024-03-25T18:24:46.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:46.506+0000 7f4221855640 1 -- >> v1:172.21.15.3:6790/0 conn(0x7f421c0b4510 legacy=0x7f421c13f550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6790/0 2024-03-25T18:24:46.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:46.506+0000 7f4222056640 1 -- >> v1:172.21.15.16:6789/0 conn(0x7f421c0b4f00 legacy=0x7f421c1423c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.16:6789/0 2024-03-25T18:24:46.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:46.907+0000 7f4221855640 1 -- >> v1:172.21.15.3:6790/0 conn(0x7f421c0b4510 legacy=0x7f421c13f550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6790/0 2024-03-25T18:24:46.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:46.907+0000 7f4222857640 1 -- >> v1:172.21.15.3:6789/0 conn(0x7f421c142ad0 legacy=0x7f421c14dc80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6789/0 2024-03-25T18:24:46.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:46.907+0000 7f4222056640 1 -- >> v1:172.21.15.16:6789/0 conn(0x7f421c0b4f00 legacy=0x7f421c1423c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.16:6789/0 2024-03-25T18:24:47.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:47.708+0000 7f4222857640 1 -- >> v1:172.21.15.3:6789/0 conn(0x7f421c142ad0 legacy=0x7f421c14dc80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6789/0 2024-03-25T18:24:47.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:47.708+0000 7f4221855640 1 -- >> v1:172.21.15.3:6790/0 conn(0x7f421c0b4510 legacy=0x7f421c13f550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6790/0 2024-03-25T18:24:47.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:47.708+0000 7f4222056640 1 -- >> v1:172.21.15.16:6789/0 conn(0x7f421c0b4f00 legacy=0x7f421c1423c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.16:6789/0 2024-03-25T18:24:49.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:49.305+0000 7f4213fff640 10 monclient(hunting): tick 2024-03-25T18:24:49.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:49.305+0000 7f4213fff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:24:49.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:49.305+0000 7f4213fff640 1 monclient(hunting): continuing hunt 2024-03-25T18:24:49.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:49.305+0000 7f4213fff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:24:49.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:49.305+0000 7f4213fff640 1 -- >> v1:172.21.15.3:6790/0 conn(0x7f421c0b4510 legacy=0x7f421c13f550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:24:49.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:49.305+0000 7f4213fff640 1 -- >> v1:172.21.15.16:6789/0 conn(0x7f421c0b4f00 legacy=0x7f421c1423c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:24:49.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:49.305+0000 7f4213fff640 1 -- >> v1:172.21.15.3:6789/0 conn(0x7f421c142ad0 legacy=0x7f421c14dc80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:24:49.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:49.305+0000 7f4213fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:49.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:49.305+0000 7f4213fff640 10 monclient(hunting): picked mon.noname-c con 0x7f4200003680 addr v1:172.21.15.16:6789/0 2024-03-25T18:24:49.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:49.305+0000 7f4213fff640 10 monclient(hunting): picked mon.noname-b con 0x7f421c0b4510 addr v1:172.21.15.3:6790/0 2024-03-25T18:24:49.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:49.305+0000 7f4213fff640 10 monclient(hunting): picked mon.noname-a con 0x7f421c142ad0 addr v1:172.21.15.3:6789/0 2024-03-25T18:24:49.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:49.305+0000 7f4213fff640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f421c13fdf0 con 0x7f421c142ad0 2024-03-25T18:24:49.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:49.305+0000 7f4213fff640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f421c0a9d90 con 0x7f4200003680 2024-03-25T18:24:49.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:49.305+0000 7f4213fff640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f421c0aa2b0 con 0x7f421c0b4510 2024-03-25T18:24:49.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:49.305+0000 7f4221855640 1 -- >> v1:172.21.15.3:6790/0 conn(0x7f421c0b4510 legacy=0x7f4200009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6790/0 2024-03-25T18:24:49.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:49.305+0000 7f4213fff640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:49.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:49.305+0000 7f4222857640 1 -- >> v1:172.21.15.3:6789/0 conn(0x7f421c142ad0 legacy=0x7f420000cb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6789/0 2024-03-25T18:24:49.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:49.305+0000 7f4222056640 1 -- >> v1:172.21.15.16:6789/0 conn(0x7f4200003680 legacy=0x7f4200005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.16:6789/0 2024-03-25T18:24:49.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:49.506+0000 7f4222857640 1 -- >> v1:172.21.15.3:6789/0 conn(0x7f421c142ad0 legacy=0x7f420000cb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6789/0 2024-03-25T18:24:49.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:49.506+0000 7f4221855640 1 -- >> v1:172.21.15.3:6790/0 conn(0x7f421c0b4510 legacy=0x7f4200009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6790/0 2024-03-25T18:24:49.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:49.506+0000 7f4222056640 1 -- >> v1:172.21.15.16:6789/0 conn(0x7f4200003680 legacy=0x7f4200005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.16:6789/0 2024-03-25T18:24:49.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:49.907+0000 7f4222857640 1 -- >> v1:172.21.15.3:6789/0 conn(0x7f421c142ad0 legacy=0x7f420000cb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6789/0 2024-03-25T18:24:49.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:49.907+0000 7f4221855640 1 -- >> v1:172.21.15.3:6790/0 conn(0x7f421c0b4510 legacy=0x7f4200009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6790/0 2024-03-25T18:24:49.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:49.907+0000 7f4222056640 1 -- >> v1:172.21.15.16:6789/0 conn(0x7f4200003680 legacy=0x7f4200005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.16:6789/0 2024-03-25T18:24:50.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:50.708+0000 7f4222857640 1 -- >> v1:172.21.15.3:6789/0 conn(0x7f421c142ad0 legacy=0x7f420000cb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6789/0 2024-03-25T18:24:50.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:50.708+0000 7f4221855640 1 -- >> v1:172.21.15.3:6790/0 conn(0x7f421c0b4510 legacy=0x7f4200009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6790/0 2024-03-25T18:24:50.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:50.708+0000 7f4222056640 1 -- >> v1:172.21.15.16:6789/0 conn(0x7f4200003680 legacy=0x7f4200005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.16:6789/0 2024-03-25T18:24:52.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:52.304+0000 7f4213fff640 10 monclient(hunting): tick 2024-03-25T18:24:52.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:52.304+0000 7f4213fff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:24:52.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:52.304+0000 7f4213fff640 1 monclient(hunting): continuing hunt 2024-03-25T18:24:52.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:52.304+0000 7f4213fff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:24:52.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:52.304+0000 7f4213fff640 1 -- >> v1:172.21.15.3:6790/0 conn(0x7f421c0b4510 legacy=0x7f4200009300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:24:52.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:52.304+0000 7f4213fff640 1 -- >> v1:172.21.15.16:6789/0 conn(0x7f4200003680 legacy=0x7f4200005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:24:52.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:52.304+0000 7f4213fff640 1 -- >> v1:172.21.15.3:6789/0 conn(0x7f421c142ad0 legacy=0x7f420000cb90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:24:52.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:52.304+0000 7f4213fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:24:52.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:52.304+0000 7f4213fff640 10 monclient(hunting): picked mon.noname-a con 0x7f421c142ad0 addr v1:172.21.15.3:6789/0 2024-03-25T18:24:52.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:52.304+0000 7f4213fff640 10 monclient(hunting): picked mon.noname-b con 0x7f421c0b4510 addr v1:172.21.15.3:6790/0 2024-03-25T18:24:52.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:52.304+0000 7f4213fff640 10 monclient(hunting): picked mon.noname-c con 0x7f4200003680 addr v1:172.21.15.16:6789/0 2024-03-25T18:24:52.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:52.304+0000 7f4213fff640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f421c13fc60 con 0x7f421c142ad0 2024-03-25T18:24:52.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:52.304+0000 7f4213fff640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f421c13fdf0 con 0x7f4200003680 2024-03-25T18:24:52.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:52.304+0000 7f4213fff640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f421c0a9d90 con 0x7f421c0b4510 2024-03-25T18:24:52.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:52.304+0000 7f4213fff640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:52.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:52.304+0000 7f4221855640 1 -- >> v1:172.21.15.3:6790/0 conn(0x7f421c0b4510 legacy=0x7f4200009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6790/0 2024-03-25T18:24:52.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:52.304+0000 7f4222056640 1 -- >> v1:172.21.15.3:6789/0 conn(0x7f421c142ad0 legacy=0x7f42000103e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6789/0 2024-03-25T18:24:52.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:52.304+0000 7f4222857640 1 -- >> v1:172.21.15.16:6789/0 conn(0x7f4200003680 legacy=0x7f4200005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.16:6789/0 2024-03-25T18:24:52.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:52.505+0000 7f4222857640 1 -- >> v1:172.21.15.16:6789/0 conn(0x7f4200003680 legacy=0x7f4200005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.16:6789/0 2024-03-25T18:24:52.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:52.505+0000 7f4221855640 1 -- >> v1:172.21.15.3:6790/0 conn(0x7f421c0b4510 legacy=0x7f4200009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6790/0 2024-03-25T18:24:52.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:52.505+0000 7f4222056640 1 -- >> v1:172.21.15.3:6789/0 conn(0x7f421c142ad0 legacy=0x7f42000103e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6789/0 2024-03-25T18:24:52.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:52.906+0000 7f4222056640 1 -- >> v1:172.21.15.3:6789/0 conn(0x7f421c142ad0 legacy=0x7f42000103e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6789/0 2024-03-25T18:24:52.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:52.906+0000 7f4221855640 1 -- >> v1:172.21.15.3:6790/0 conn(0x7f421c0b4510 legacy=0x7f4200009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6790/0 2024-03-25T18:24:52.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:52.906+0000 7f4222857640 1 -- >> v1:172.21.15.16:6789/0 conn(0x7f4200003680 legacy=0x7f4200005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.16:6789/0 2024-03-25T18:24:53.654 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:00:07.346 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:53.663 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:00:07.393 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:53.668 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:00:07.398 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:53.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:53.707+0000 7f4222056640 1 -- >> v1:172.21.15.3:6789/0 conn(0x7f421c142ad0 legacy=0x7f42000103e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6789/0 2024-03-25T18:24:53.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:53.707+0000 7f4221855640 1 -- >> v1:172.21.15.3:6790/0 conn(0x7f421c0b4510 legacy=0x7f4200009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.3:6790/0 2024-03-25T18:24:53.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:53.707+0000 7f4222857640 1 -- >> v1:172.21.15.16:6789/0 conn(0x7f4200003680 legacy=0x7f4200005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.16:6789/0 2024-03-25T18:24:53.980 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:00:07.672 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:53.990 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:00:07.719 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:53.995 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:00:07.724 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:55.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:55.304+0000 7f4213fff640 10 monclient(hunting): tick 2024-03-25T18:24:55.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:55.304+0000 7f4213fff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:24:55.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:55.304+0000 7f4213fff640 1 monclient(hunting): continuing hunt 2024-03-25T18:24:55.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:55.304+0000 7f4213fff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:24:55.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:55.304+0000 7f4213fff640 1 -- >> v1:172.21.15.3:6790/0 conn(0x7f421c0b4510 legacy=0x7f4200009300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:24:55.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:55.304+0000 7f4213fff640 1 -- >> v1:172.21.15.16:6789/0 conn(0x7f4200003680 legacy=0x7f4200005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:24:55.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:55.304+0000 7f4213fff640 1 -- >> v1:172.21.15.3:6789/0 conn(0x7f421c142ad0 legacy=0x7f42000103e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:24:55.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:55.304+0000 7f4213fff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:55.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:55.304+0000 7f4213fff640 10 monclient(hunting): picked mon.noname-b con 0x7f421c0b4510 addr v1:172.21.15.3:6790/0 2024-03-25T18:24:55.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:55.304+0000 7f4213fff640 10 monclient(hunting): picked mon.noname-c con 0x7f4200003680 addr v1:172.21.15.16:6789/0 2024-03-25T18:24:55.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:55.304+0000 7f4213fff640 10 monclient(hunting): picked mon.noname-a con 0x7f421c142ad0 addr v1:172.21.15.3:6789/0 2024-03-25T18:24:55.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:55.305+0000 7f4213fff640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f421c0aa2b0 con 0x7f421c142ad0 2024-03-25T18:24:55.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:55.305+0000 7f4213fff640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f421c13fc60 con 0x7f4200003680 2024-03-25T18:24:55.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:55.305+0000 7f4213fff640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f421c13fdf0 con 0x7f421c0b4510 2024-03-25T18:24:55.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:55.305+0000 7f4213fff640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:55.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:55.517+0000 7f4221855640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4200003680 0x7f4200005ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:57954/0 (socket says 172.21.15.3:57954) 2024-03-25T18:24:55.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:55.517+0000 7f4221855640 1 -- 172.21.15.3:0/2811703338 learned_addr learned my addr 172.21.15.3:0/2811703338 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:24:55.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:55.699+0000 7f4222056640 1 -- 172.21.15.3:0/2811703338 >> v1:172.21.15.3:6790/0 conn(0x7f421c0b4510 legacy=0x7f4200010af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-25T18:24:55.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:55.699+0000 7f4222056640 1 -- 172.21.15.3:0/2811703338 >> v1:172.21.15.3:6790/0 conn(0x7f421c0b4510 legacy=0x7f4200010af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T18:24:55.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:55.699+0000 7f4222056640 1 --1- 172.21.15.3:0/2811703338 >> v1:172.21.15.3:6790/0 conn(0x7f421c0b4510 0x7f4200010af0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T18:24:55.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:55.706+0000 7f4221855640 1 -- 172.21.15.3:0/2811703338 >> v1:172.21.15.16:6789/0 conn(0x7f4200003680 legacy=0x7f4200005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-25T18:24:55.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:55.706+0000 7f4221855640 1 -- 172.21.15.3:0/2811703338 >> v1:172.21.15.16:6789/0 conn(0x7f4200003680 legacy=0x7f4200005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T18:24:55.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:55.706+0000 7f4221855640 1 --1- 172.21.15.3:0/2811703338 >> v1:172.21.15.16:6789/0 conn(0x7f4200003680 0x7f4200005ad0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T18:24:55.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:55.718+0000 7f4222857640 1 -- 172.21.15.3:0/2811703338 >> v1:172.21.15.3:6789/0 conn(0x7f421c142ad0 legacy=0x7f42000103e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-25T18:24:55.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:55.718+0000 7f4222857640 1 -- 172.21.15.3:0/2811703338 >> v1:172.21.15.3:6789/0 conn(0x7f421c142ad0 legacy=0x7f42000103e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T18:24:55.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:55.718+0000 7f4222857640 1 --1- 172.21.15.3:0/2811703338 >> v1:172.21.15.3:6789/0 conn(0x7f421c142ad0 0x7f42000103e0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T18:24:58.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:58.305+0000 7f4213fff640 10 monclient(hunting): tick 2024-03-25T18:24:58.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:58.305+0000 7f4213fff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:24:58.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:58.305+0000 7f4213fff640 1 monclient(hunting): continuing hunt 2024-03-25T18:24:58.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:58.305+0000 7f4213fff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:24:58.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:58.305+0000 7f4213fff640 1 -- 172.21.15.3:0/2811703338 >> v1:172.21.15.3:6790/0 conn(0x7f421c0b4510 legacy=0x7f4200010af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:58.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:58.305+0000 7f4213fff640 1 -- 172.21.15.3:0/2811703338 >> v1:172.21.15.16:6789/0 conn(0x7f4200003680 legacy=0x7f4200005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:58.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:58.305+0000 7f4213fff640 1 -- 172.21.15.3:0/2811703338 >> v1:172.21.15.3:6789/0 conn(0x7f421c142ad0 legacy=0x7f42000103e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:58.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:58.305+0000 7f4213fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:24:58.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:58.305+0000 7f4213fff640 10 monclient(hunting): picked mon.noname-a con 0x7f420000cc00 addr v1:172.21.15.3:6789/0 2024-03-25T18:24:58.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:58.305+0000 7f4213fff640 10 monclient(hunting): picked mon.noname-c con 0x7f4200003680 addr v1:172.21.15.16:6789/0 2024-03-25T18:24:58.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:58.305+0000 7f4213fff640 10 monclient(hunting): picked mon.noname-b con 0x7f421c0b4510 addr v1:172.21.15.3:6790/0 2024-03-25T18:24:58.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:58.305+0000 7f4213fff640 1 -- 172.21.15.3:0/2811703338 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f421c0a9d90 con 0x7f420000cc00 2024-03-25T18:24:58.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:58.305+0000 7f4213fff640 1 -- 172.21.15.3:0/2811703338 --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f420000e0a0 con 0x7f4200003680 2024-03-25T18:24:58.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:58.305+0000 7f4213fff640 1 -- 172.21.15.3:0/2811703338 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42000103e0 con 0x7f421c0b4510 2024-03-25T18:24:58.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:24:58.305+0000 7f4213fff640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:01.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:01.305+0000 7f4213fff640 10 monclient(hunting): tick 2024-03-25T18:25:01.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:01.305+0000 7f4213fff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:25:01.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:01.305+0000 7f4213fff640 1 monclient(hunting): continuing hunt 2024-03-25T18:25:01.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:01.305+0000 7f4213fff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:25:01.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:01.305+0000 7f4213fff640 1 -- 172.21.15.3:0/2811703338 >> v1:172.21.15.3:6790/0 conn(0x7f421c0b4510 legacy=0x7f4200010af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:01.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:01.305+0000 7f4213fff640 1 -- 172.21.15.3:0/2811703338 >> v1:172.21.15.16:6789/0 conn(0x7f4200003680 legacy=0x7f4200005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:01.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:01.305+0000 7f4213fff640 1 -- 172.21.15.3:0/2811703338 >> v1:172.21.15.3:6789/0 conn(0x7f420000cc00 legacy=0x7f42000092e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:01.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:01.305+0000 7f4213fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:01.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:01.305+0000 7f4213fff640 10 monclient(hunting): picked mon.noname-b con 0x7f420000d040 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:01.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:01.305+0000 7f4213fff640 10 monclient(hunting): picked mon.noname-a con 0x7f420000db90 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:01.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:01.305+0000 7f4213fff640 10 monclient(hunting): picked mon.noname-c con 0x7f4200003680 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:01.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:01.305+0000 7f4213fff640 1 -- 172.21.15.3:0/2811703338 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4200010610 con 0x7f420000db90 2024-03-25T18:25:01.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:01.305+0000 7f4213fff640 1 -- 172.21.15.3:0/2811703338 --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4200006200 con 0x7f4200003680 2024-03-25T18:25:01.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:01.305+0000 7f4213fff640 1 -- 172.21.15.3:0/2811703338 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4200006390 con 0x7f420000d040 2024-03-25T18:25:01.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:01.305+0000 7f4213fff640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:04.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.305+0000 7f4213fff640 10 monclient(hunting): tick 2024-03-25T18:25:04.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.305+0000 7f4213fff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:25:04.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.305+0000 7f4213fff640 1 monclient(hunting): continuing hunt 2024-03-25T18:25:04.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.305+0000 7f4213fff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:25:04.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.305+0000 7f4213fff640 1 -- 172.21.15.3:0/2811703338 >> v1:172.21.15.3:6790/0 conn(0x7f420000d040 legacy=0x7f420000d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:04.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.305+0000 7f4213fff640 1 -- 172.21.15.3:0/2811703338 >> v1:172.21.15.16:6789/0 conn(0x7f4200003680 legacy=0x7f4200010a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:04.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.305+0000 7f4213fff640 1 -- 172.21.15.3:0/2811703338 >> v1:172.21.15.3:6789/0 conn(0x7f420000db90 legacy=0x7f42000154e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:04.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.305+0000 7f4213fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:04.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.305+0000 7f4213fff640 10 monclient(hunting): picked mon.noname-a con 0x7f42000095c0 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:04.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.305+0000 7f4213fff640 10 monclient(hunting): picked mon.noname-c con 0x7f4200006d30 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:04.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.305+0000 7f4213fff640 10 monclient(hunting): picked mon.noname-b con 0x7f420000d040 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:04.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.305+0000 7f4213fff640 1 -- 172.21.15.3:0/2811703338 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4200009390 con 0x7f42000095c0 2024-03-25T18:25:04.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.305+0000 7f4213fff640 1 -- 172.21.15.3:0/2811703338 --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4200010af0 con 0x7f4200006d30 2024-03-25T18:25:04.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.305+0000 7f4213fff640 1 -- 172.21.15.3:0/2811703338 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4200010d20 con 0x7f420000d040 2024-03-25T18:25:04.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.305+0000 7f4213fff640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:04.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.330+0000 7f4221855640 1 -- 172.21.15.3:0/2811703338 >> v1:172.21.15.16:6789/0 conn(0x7f4200006d30 legacy=0x7f4200005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-25T18:25:04.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.330+0000 7f4221855640 1 -- 172.21.15.3:0/2811703338 >> v1:172.21.15.16:6789/0 conn(0x7f4200006d30 legacy=0x7f4200005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:25:04.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.330+0000 7f4221855640 1 --1- 172.21.15.3:0/2811703338 >> v1:172.21.15.16:6789/0 conn(0x7f4200006d30 0x7f4200005ab0 :-1 s=OPENED pgs=20 cs=1 l=1).handle_message read tag failed 2024-03-25T18:25:04.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.330+0000 7f4221855640 1 --1- 172.21.15.3:0/2811703338 >> v1:172.21.15.16:6789/0 conn(0x7f4200006d30 0x7f4200005ab0 :-1 s=OPENED pgs=20 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:25:04.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.330+0000 7f4221054640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.16:6789/0 2024-03-25T18:25:04.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.566+0000 7f4221054640 1 -- 172.21.15.3:0/2811703338 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 690682490 0 0) 0x7f4200009390 con 0x7f42000095c0 2024-03-25T18:25:04.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.566+0000 7f4221054640 10 monclient(hunting): _negotiate 2024-03-25T18:25:04.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.566+0000 7f4221054640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:04.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.566+0000 7f4221054640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:04.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.566+0000 7f4221054640 10 monclient(hunting): my global_id is 4104 2024-03-25T18:25:04.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.566+0000 7f4221054640 1 -- 172.21.15.3:0/2811703338 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4208003520 con 0x7f42000095c0 2024-03-25T18:25:04.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.610+0000 7f4221054640 1 -- 172.21.15.3:0/2811703338 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 942607479 0 0) 0x7f4200010d20 con 0x7f420000d040 2024-03-25T18:25:04.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.610+0000 7f4221054640 10 monclient(hunting): _negotiate 2024-03-25T18:25:04.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.610+0000 7f4221054640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:04.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.610+0000 7f4221054640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:04.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.610+0000 7f4221054640 10 monclient(hunting): my global_id is 4106 2024-03-25T18:25:04.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.610+0000 7f4221054640 1 -- 172.21.15.3:0/2811703338 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4200009390 con 0x7f420000d040 2024-03-25T18:25:04.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.636+0000 7f4221054640 1 -- 172.21.15.3:0/2811703338 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3576891082 0 0) 0x7f4208003520 con 0x7f42000095c0 2024-03-25T18:25:04.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.636+0000 7f4221054640 1 -- 172.21.15.3:0/2811703338 >> v1:172.21.15.3:6790/0 conn(0x7f420000d040 legacy=0x7f420000d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:04.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.636+0000 7f4221054640 1 -- 172.21.15.3:0/2811703338 >> v1:172.21.15.16:6789/0 conn(0x7f4200006d30 legacy=0x7f4200005ab0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:25:04.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.636+0000 7f4221054640 10 monclient: _finish_hunting 0 2024-03-25T18:25:04.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.636+0000 7f4221054640 1 monclient: found mon.noname-a 2024-03-25T18:25:04.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.636+0000 7f4221054640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:04.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.636+0000 7f4221054640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:25:04.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.636+0000 7f4221054640 1 -- 172.21.15.3:0/2811703338 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4200010f50 con 0x7f42000095c0 2024-03-25T18:25:04.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.636+0000 7f4221054640 10 monclient: _finish_auth 0 2024-03-25T18:25:04.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.636+0000 7f4221054640 10 monclient: _check_auth_tickets 2024-03-25T18:25:04.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.636+0000 7f4221054640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:04.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.636+0000 7f4221054640 1 -- 172.21.15.3:0/2811703338 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4218005040 con 0x7f42000095c0 2024-03-25T18:25:04.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.636+0000 7f42242ba640 5 monclient: authenticate success, global_id 4104 2024-03-25T18:25:04.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.636+0000 7f42242ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:04.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.636+0000 7f4221054640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:04.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.636+0000 7f4221054640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:04.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.636+0000 7f4221054640 10 monclient: dump: 2024-03-25T18:25:04.640 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-03-25T18:25:04.640 INFO:teuthology.orchestra.run.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:04.640 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:04.640 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:04.640 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:04.640 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:04.640 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:04.640 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:04.640 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:04.640 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-25T18:25:04.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.636+0000 7f42242ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:04.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.704+0000 7f4221054640 1 -- 172.21.15.3:0/2811703338 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42180051a0 con 0x7f42000095c0 2024-03-25T18:25:04.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.704+0000 7f4221054640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:04.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.704+0000 7f42242ba640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:04.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.704+0000 7f42242ba640 4 set_mon_vals no callback set 2024-03-25T18:25:04.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.705+0000 7f42242ba640 10 monclient: shutdown 2024-03-25T18:25:04.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.705+0000 7f42242ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:04.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.705+0000 7f42242ba640 1 -- 172.21.15.3:0/2811703338 >> v1:172.21.15.3:6789/0 conn(0x7f42000095c0 legacy=0x7f4200006520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:04.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.705+0000 7f42242ba640 1 -- 172.21.15.3:0/2811703338 shutdown_connections 2024-03-25T18:25:04.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.705+0000 7f42242ba640 1 -- 172.21.15.3:0/2811703338 >> 172.21.15.3:0/2811703338 conn(0x7f421c066b00 msgr2=0x7f421c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:04.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.705+0000 7f42242ba640 1 -- 172.21.15.3:0/2811703338 shutdown_connections 2024-03-25T18:25:04.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.705+0000 7f42242ba640 1 -- 172.21.15.3:0/2811703338 wait complete. 2024-03-25T18:25:04.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.705+0000 7f42242ba640 10 monclient: build_initial_monmap 2024-03-25T18:25:04.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.705+0000 7f42242ba640 10 monclient: monmap: 2024-03-25T18:25:04.709 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-03-25T18:25:04.709 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:04.709 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-03-25T18:25:04.709 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-03-25T18:25:04.709 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:04.709 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-03-25T18:25:04.709 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:04.709 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:25:04.709 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:25:04.709 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-25T18:25:04.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.706+0000 7f42242ba640 1 librados: starting msgr at 2024-03-25T18:25:04.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.706+0000 7f42242ba640 1 librados: starting objecter 2024-03-25T18:25:04.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.706+0000 7f42242ba640 1 Processor -- start 2024-03-25T18:25:04.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.706+0000 7f42242ba640 1 -- start start 2024-03-25T18:25:04.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.706+0000 7f42242ba640 1 librados: setting wanted keys 2024-03-25T18:25:04.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.706+0000 7f42242ba640 1 librados: calling monclient init 2024-03-25T18:25:04.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.706+0000 7f42242ba640 10 monclient: init 2024-03-25T18:25:04.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.706+0000 7f42242ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:04.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.706+0000 7f42242ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:04.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.706+0000 7f42242ba640 10 monclient(hunting): picked mon.noname-b con 0x7f4200006d30 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:04.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.706+0000 7f42242ba640 10 monclient(hunting): picked mon.noname-c con 0x7f42000095c0 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:04.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.706+0000 7f42242ba640 10 monclient(hunting): picked mon.noname-a con 0x7f420000d040 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:04.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.706+0000 7f42242ba640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f421c145160 con 0x7f420000d040 2024-03-25T18:25:04.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.706+0000 7f42242ba640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f421c0ad8e0 con 0x7f4200006d30 2024-03-25T18:25:04.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.706+0000 7f42242ba640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f421c0ad370 con 0x7f42000095c0 2024-03-25T18:25:04.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.706+0000 7f42242ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:04.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.706+0000 7f42242ba640 10 monclient(hunting): authenticate will time out at 7358.967256s 2024-03-25T18:25:04.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.707+0000 7f4221855640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f42000095c0 0x7f421c145610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:46774/0 (socket says 172.21.15.3:46774) 2024-03-25T18:25:04.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.707+0000 7f4221855640 1 -- 172.21.15.3:0/2298424249 learned_addr learned my addr 172.21.15.3:0/2298424249 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:25:04.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.707+0000 7f4212ffd640 10 client.?.objecter ms_handle_connect 0x7f42000095c0 2024-03-25T18:25:04.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.707+0000 7f4212ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:04.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.710+0000 7f4212ffd640 10 client.?.objecter ms_handle_connect 0x7f4200006d30 2024-03-25T18:25:04.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.710+0000 7f4212ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:04.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.763+0000 7f4212ffd640 10 client.?.objecter ms_handle_connect 0x7f420000d040 2024-03-25T18:25:04.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.763+0000 7f4212ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:04.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.777+0000 7f4212ffd640 1 -- 172.21.15.3:0/2298424249 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3246439929 0 0) 0x7f421c0ad8e0 con 0x7f4200006d30 2024-03-25T18:25:04.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.777+0000 7f4212ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:04.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.777+0000 7f4212ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:04.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.777+0000 7f4212ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:04.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.777+0000 7f4212ffd640 10 monclient(hunting): my global_id is 4105 2024-03-25T18:25:04.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.777+0000 7f4212ffd640 1 -- 172.21.15.3:0/2298424249 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42000112c0 con 0x7f4200006d30 2024-03-25T18:25:04.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.782+0000 7f4212ffd640 1 -- 172.21.15.3:0/2298424249 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2505628928 0 0) 0x7f421c0ad370 con 0x7f42000095c0 2024-03-25T18:25:04.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.782+0000 7f4212ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:04.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.782+0000 7f4212ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:04.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.782+0000 7f4212ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:04.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.782+0000 7f4212ffd640 10 monclient(hunting): my global_id is 4109 2024-03-25T18:25:04.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.782+0000 7f4212ffd640 1 -- 172.21.15.3:0/2298424249 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f421c0ad8e0 con 0x7f42000095c0 2024-03-25T18:25:04.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.787+0000 7f4212ffd640 1 -- 172.21.15.3:0/2298424249 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1897982213 0 0) 0x7f421c145160 con 0x7f420000d040 2024-03-25T18:25:04.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.787+0000 7f4212ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:04.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.787+0000 7f4212ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:04.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.787+0000 7f4212ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:04.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.787+0000 7f4212ffd640 10 monclient(hunting): my global_id is 4107 2024-03-25T18:25:04.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.787+0000 7f4212ffd640 1 -- 172.21.15.3:0/2298424249 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f421c0ad370 con 0x7f420000d040 2024-03-25T18:25:04.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.828+0000 7f4212ffd640 1 -- 172.21.15.3:0/2298424249 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 645382539 0 0) 0x7f421c0ad370 con 0x7f420000d040 2024-03-25T18:25:04.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.828+0000 7f4212ffd640 1 -- 172.21.15.3:0/2298424249 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f421c145160 con 0x7f420000d040 2024-03-25T18:25:04.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.828+0000 7f4212ffd640 1 -- 172.21.15.3:0/2298424249 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f42140044a0 con 0x7f420000d040 2024-03-25T18:25:04.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.828+0000 7f4212ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:04.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.828+0000 7f4212ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:04.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.828+0000 7f4212ffd640 10 monclient(hunting): dump: 2024-03-25T18:25:04.829 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-03-25T18:25:04.829 INFO:teuthology.orchestra.run.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:04.829 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:04.830 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:04.830 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:04.830 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:04.830 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:04.830 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:04.830 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:04.830 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-25T18:25:04.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.828+0000 7f4212ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:04.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.898+0000 7f4212ffd640 1 -- 172.21.15.3:0/2298424249 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 357433854 0 0) 0x7f421c145160 con 0x7f420000d040 2024-03-25T18:25:04.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.898+0000 7f4212ffd640 1 -- 172.21.15.3:0/2298424249 >> v1:172.21.15.3:6790/0 conn(0x7f42000095c0 legacy=0x7f421c145610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:04.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.898+0000 7f4212ffd640 1 -- 172.21.15.3:0/2298424249 >> v1:172.21.15.16:6789/0 conn(0x7f4200006d30 legacy=0x7f421c144a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:04.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.898+0000 7f4212ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:04.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.898+0000 7f4212ffd640 1 monclient: found mon.a 2024-03-25T18:25:04.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.898+0000 7f4212ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:04.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.898+0000 7f4212ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:04.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.898+0000 7f4212ffd640 1 -- 172.21.15.3:0/2298424249 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f421c0ad500 con 0x7f420000d040 2024-03-25T18:25:04.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.898+0000 7f4212ffd640 10 monclient: _finish_auth 0 2024-03-25T18:25:04.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.898+0000 7f4212ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:04.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.898+0000 7f4212ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:04.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.898+0000 7f42242ba640 5 monclient: authenticate success, global_id 4107 2024-03-25T18:25:04.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.898+0000 7f42242ba640 10 monclient: _renew_subs 2024-03-25T18:25:04.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.898+0000 7f42242ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:04.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.898+0000 7f42242ba640 1 -- 172.21.15.3:0/2298424249 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f421c0b0e30 con 0x7f420000d040 2024-03-25T18:25:04.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.898+0000 7f42242ba640 10 client.4107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:04.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.898+0000 7f42242ba640 10 monclient: _renew_subs 2024-03-25T18:25:04.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.898+0000 7f42242ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:04.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.898+0000 7f42242ba640 1 -- 172.21.15.3:0/2298424249 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f421c1db770 con 0x7f420000d040 2024-03-25T18:25:04.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.898+0000 7f42242ba640 1 librados: init done 2024-03-25T18:25:04.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.899+0000 7f42242ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:04.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.899+0000 7f42242ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:04.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.899+0000 7f42242ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:04.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:04.899+0000 7f42242ba640 1 -- 172.21.15.3:0/2298424249 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f421c0aeac0 con 0x7f420000d040 2024-03-25T18:25:05.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.118+0000 7f4212ffd640 1 -- 172.21.15.3:0/2298424249 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42140048b0 con 0x7f420000d040 2024-03-25T18:25:05.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.118+0000 7f4212ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:05.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.118+0000 7f4212ffd640 1 -- 172.21.15.3:0/2298424249 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4214004cc0 con 0x7f420000d040 2024-03-25T18:25:05.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.118+0000 7f4212ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:05.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.118+0000 7f4212ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:05.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.118+0000 7f4212ffd640 10 monclient: dump: 2024-03-25T18:25:05.120 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-03-25T18:25:05.120 INFO:teuthology.orchestra.run.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:05.120 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:05.120 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:05.120 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:05.120 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:05.120 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:05.120 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:05.121 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:05.121 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-25T18:25:05.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.118+0000 7f4220853640 4 set_mon_vals no callback set 2024-03-25T18:25:05.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.118+0000 7f4212ffd640 1 -- 172.21.15.3:0/2298424249 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4214005270 con 0x7f420000d040 2024-03-25T18:25:05.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.118+0000 7f4212ffd640 1 -- 172.21.15.3:0/2298424249 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3459837783 0 0) 0x7f4214006740 con 0x7f420000d040 2024-03-25T18:25:05.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.118+0000 7f4212ffd640 10 client.4107.objecter ms_dispatch 0x7f421c000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T18:25:05.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.118+0000 7f4212ffd640 3 client.4107.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T18:25:05.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.118+0000 7f4212ffd640 3 client.4107.objecter handle_osd_map decoding full epoch 1 2024-03-25T18:25:05.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.118+0000 7f4212ffd640 20 client.4107.objecter dump_active .. 0 homeless 2024-03-25T18:25:05.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.118+0000 7f4212ffd640 1 -- 172.21.15.3:0/2298424249 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4214006b10 con 0x7f420000d040 2024-03-25T18:25:05.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.118+0000 7f4212ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:05.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.118+0000 7f4212ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:05.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.188+0000 7f42242ba640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:25:05.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.188+0000 7f42242ba640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:25:05.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.188+0000 7f42242ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:05.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.188+0000 7f42242ba640 1 -- 172.21.15.3:0/2298424249 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f421c13e0f0 con 0x7f420000d040 2024-03-25T18:25:05.369 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-25T18:25:05.368+0000 7f0c86a58040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:25:05.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.410+0000 7f4212ffd640 1 -- 172.21.15.3:0/2298424249 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f4214005600 con 0x7f420000d040 2024-03-25T18:25:05.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.410+0000 7f4212ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:25:05.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.410+0000 7f4212ffd640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T18:25:05.412 INFO:teuthology.orchestra.run.smithi003.stderr:adjusted tunables profile to default 2024-03-25T18:25:05.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.411+0000 7f42242ba640 10 librados: watch_flush enter 2024-03-25T18:25:05.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.411+0000 7f42242ba640 10 librados: watch_flush exit 2024-03-25T18:25:05.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.411+0000 7f42242ba640 20 client.4107.objecter shutdown clearing up homeless session... 2024-03-25T18:25:05.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.411+0000 7f42242ba640 10 client.4107.objecter successfully canceled tick 2024-03-25T18:25:05.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.411+0000 7f42242ba640 10 monclient: shutdown 2024-03-25T18:25:05.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.411+0000 7f42242ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:05.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.411+0000 7f42242ba640 1 -- 172.21.15.3:0/2298424249 >> v1:172.21.15.3:6789/0 conn(0x7f420000d040 legacy=0x7f421c1e2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:05.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.411+0000 7f42242ba640 1 -- 172.21.15.3:0/2298424249 shutdown_connections 2024-03-25T18:25:05.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.411+0000 7f42242ba640 1 -- 172.21.15.3:0/2298424249 >> 172.21.15.3:0/2298424249 conn(0x7f421c066b00 msgr2=0x7f421c13e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:05.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.411+0000 7f42242ba640 1 -- 172.21.15.3:0/2298424249 shutdown_connections 2024-03-25T18:25:05.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.411+0000 7f42242ba640 1 -- 172.21.15.3:0/2298424249 wait complete. 2024-03-25T18:25:05.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:05.411+0000 7f42242ba640 1 librados: shutdown 2024-03-25T18:25:05.423 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T18:25:05.423 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T18:25:05.424 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-25T18:25:05.424 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T18:25:05.441 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-25T18:25:05.440+0000 7f0c86a58040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:25:05.451 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-25T18:25:05.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T18:25:05.520 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-25T18:25:05.520 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T18:25:05.566 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-03-25T18:25:05.565+0000 7f49c1ff1040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:25:05.587 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-25T18:25:05.587 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T18:25:05.621 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-03-25T18:25:05.621+0000 7f49c1ff1040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:25:05.653 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-25T18:25:05.653 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T18:25:05.679 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-25T18:25:05.680 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T18:25:05.746 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-25T18:25:05.746 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T18:25:05.812 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-25T18:25:05.812 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T18:25:05.878 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd new d8dee66f-c1ad-477e-8d93-f1bf6ec49cac 0 2024-03-25T18:25:05.995 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-25T18:25:05.994+0000 7f0c86a58040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:25:06.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.017+0000 7f08b52d9640 10 monclient: get_monmap_and_config 2024-03-25T18:25:06.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.017+0000 7f08b52d9640 10 monclient: build_initial_monmap 2024-03-25T18:25:06.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.017+0000 7f08b52d9640 10 monclient: monmap: 2024-03-25T18:25:06.019 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 0 2024-03-25T18:25:06.019 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:06.019 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:25:06.019033+0000 2024-03-25T18:25:06.019 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:25:06.019033+0000 2024-03-25T18:25:06.019 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:06.019 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:06.019 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:06.019 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:25:06.019 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:25:06.019 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:06.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.018+0000 7f08b52d9640 1 Processor -- start 2024-03-25T18:25:06.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.018+0000 7f08b52d9640 1 -- start start 2024-03-25T18:25:06.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.018+0000 7f08b52d9640 10 monclient: init 2024-03-25T18:25:06.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.019+0000 7f08b52d9640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:06.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.019+0000 7f08b52d9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:06.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.019+0000 7f08b52d9640 10 monclient(hunting): picked mon.noname-c con 0x7f08b00b4f00 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:06.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.019+0000 7f08b52d9640 10 monclient(hunting): picked mon.noname-b con 0x7f08b00b4510 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:06.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.019+0000 7f08b52d9640 10 monclient(hunting): picked mon.noname-a con 0x7f08b0142ad0 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:06.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.019+0000 7f08b52d9640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b00a9d90 con 0x7f08b0142ad0 2024-03-25T18:25:06.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.019+0000 7f08b52d9640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b00aa2b0 con 0x7f08b00b4f00 2024-03-25T18:25:06.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.019+0000 7f08b52d9640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b013fc60 con 0x7f08b00b4510 2024-03-25T18:25:06.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.019+0000 7f08b52d9640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:06.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.019+0000 7f08b52d9640 10 monclient(hunting): authenticate will time out at 7363.154841s 2024-03-25T18:25:06.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.019+0000 7f08aeffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f08b00b4f00 0x7f08b01423c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37722/0 (socket says 172.21.15.16:37722) 2024-03-25T18:25:06.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.019+0000 7f08aeffd640 1 -- 172.21.15.16:0/3242212636 learned_addr learned my addr 172.21.15.16:0/3242212636 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-25T18:25:06.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.022+0000 7f08adffb640 1 -- 172.21.15.16:0/3242212636 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1617338188 0 0) 0x7f08b00aa2b0 con 0x7f08b00b4f00 2024-03-25T18:25:06.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.022+0000 7f08adffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:06.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.022+0000 7f08adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:06.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.022+0000 7f08adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:06.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.022+0000 7f08adffb640 10 monclient(hunting): my global_id is 4111 2024-03-25T18:25:06.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.022+0000 7f08adffb640 1 -- 172.21.15.16:0/3242212636 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0898003520 con 0x7f08b00b4f00 2024-03-25T18:25:06.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.022+0000 7f08adffb640 1 -- 172.21.15.16:0/3242212636 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 498921957 0 0) 0x7f08b013fc60 con 0x7f08b00b4510 2024-03-25T18:25:06.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.022+0000 7f08adffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:06.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.022+0000 7f08adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:06.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.022+0000 7f08adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:06.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.022+0000 7f08adffb640 10 monclient(hunting): my global_id is 4115 2024-03-25T18:25:06.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.022+0000 7f08adffb640 1 -- 172.21.15.16:0/3242212636 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08b00aa2b0 con 0x7f08b00b4510 2024-03-25T18:25:06.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.023+0000 7f08adffb640 1 -- 172.21.15.16:0/3242212636 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3242125044 0 0) 0x7f08b00a9d90 con 0x7f08b0142ad0 2024-03-25T18:25:06.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.023+0000 7f08adffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:06.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.023+0000 7f08adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:06.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.023+0000 7f08adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:06.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.023+0000 7f08adffb640 10 monclient(hunting): my global_id is 4116 2024-03-25T18:25:06.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.023+0000 7f08adffb640 1 -- 172.21.15.16:0/3242212636 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08b013fc60 con 0x7f08b0142ad0 2024-03-25T18:25:06.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.025+0000 7f08adffb640 1 -- 172.21.15.16:0/3242212636 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3645592786 0 0) 0x7f0898003520 con 0x7f08b00b4f00 2024-03-25T18:25:06.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.025+0000 7f08adffb640 1 -- 172.21.15.16:0/3242212636 >> v1:172.21.15.3:6790/0 conn(0x7f08b00b4510 legacy=0x7f08b013f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:06.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.025+0000 7f08adffb640 1 -- 172.21.15.16:0/3242212636 >> v1:172.21.15.3:6789/0 conn(0x7f08b0142ad0 legacy=0x7f08b014dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:06.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.025+0000 7f08adffb640 10 monclient: _finish_hunting 0 2024-03-25T18:25:06.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.025+0000 7f08adffb640 1 monclient: found mon.noname-c 2024-03-25T18:25:06.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.025+0000 7f08adffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:06.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.025+0000 7f08adffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:25:06.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.025+0000 7f08adffb640 1 -- 172.21.15.16:0/3242212636 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b013fdf0 con 0x7f08b00b4f00 2024-03-25T18:25:06.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.025+0000 7f08adffb640 10 monclient: _finish_auth 0 2024-03-25T18:25:06.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.025+0000 7f08adffb640 10 monclient: _check_auth_tickets 2024-03-25T18:25:06.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.025+0000 7f08adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:06.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.025+0000 7f08adffb640 1 -- 172.21.15.16:0/3242212636 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f08a4001690 con 0x7f08b00b4f00 2024-03-25T18:25:06.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.025+0000 7f08b52d9640 5 monclient: authenticate success, global_id 4111 2024-03-25T18:25:06.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.025+0000 7f08b52d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:06.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.025+0000 7f08adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:06.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.025+0000 7f08adffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:06.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.025+0000 7f08adffb640 10 monclient: dump: 2024-03-25T18:25:06.028 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:06.028 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:06.028 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:06.028 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:06.028 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:06.028 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:06.028 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:06.028 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:06.028 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:06.028 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:06.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.025+0000 7f08b52d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:06.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.052+0000 7f08adffb640 1 -- 172.21.15.16:0/3242212636 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08a40017f0 con 0x7f08b00b4f00 2024-03-25T18:25:06.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.052+0000 7f08adffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:06.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.052+0000 7f08adffb640 1 -- 172.21.15.16:0/3242212636 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f08a4001c00 con 0x7f08b00b4f00 2024-03-25T18:25:06.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.052+0000 7f08adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:06.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.052+0000 7f08adffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:06.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.052+0000 7f08adffb640 10 monclient: dump: 2024-03-25T18:25:06.053 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:06.053 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:06.053 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:06.054 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:06.054 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:06.054 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:06.054 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:06.054 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:06.054 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:06.054 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:06.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.052+0000 7f08b52d9640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:06.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.052+0000 7f08b52d9640 4 set_mon_vals no callback set 2024-03-25T18:25:06.055 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-25T18:25:06.052+0000 7f0c86a58040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:25:06.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.053+0000 7f08b52d9640 10 monclient: shutdown 2024-03-25T18:25:06.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.053+0000 7f08b52d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:06.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.053+0000 7f08b52d9640 1 -- 172.21.15.16:0/3242212636 >> v1:172.21.15.16:6789/0 conn(0x7f08b00b4f00 legacy=0x7f08b01423c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:06.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.053+0000 7f08b52d9640 1 -- 172.21.15.16:0/3242212636 shutdown_connections 2024-03-25T18:25:06.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.053+0000 7f08b52d9640 1 -- 172.21.15.16:0/3242212636 >> 172.21.15.16:0/3242212636 conn(0x7f08b0066b00 msgr2=0x7f08b0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:06.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.053+0000 7f08b52d9640 1 -- 172.21.15.16:0/3242212636 shutdown_connections 2024-03-25T18:25:06.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.053+0000 7f08b52d9640 1 -- 172.21.15.16:0/3242212636 wait complete. 2024-03-25T18:25:06.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.054+0000 7f08b52d9640 10 monclient: build_initial_monmap 2024-03-25T18:25:06.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.054+0000 7f08b52d9640 10 monclient: monmap: 2024-03-25T18:25:06.057 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 0 2024-03-25T18:25:06.058 INFO:teuthology.orchestra.run.smithi016.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:06.058 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 0.000000 2024-03-25T18:25:06.058 INFO:teuthology.orchestra.run.smithi016.stderr:created 0.000000 2024-03-25T18:25:06.058 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:06.058 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 1 2024-03-25T18:25:06.058 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:06.058 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:25:06.058 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:25:06.058 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:06.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.054+0000 7f08b52d9640 1 librados: starting msgr at 2024-03-25T18:25:06.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.054+0000 7f08b52d9640 1 librados: starting objecter 2024-03-25T18:25:06.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.054+0000 7f08b52d9640 1 Processor -- start 2024-03-25T18:25:06.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.054+0000 7f08b52d9640 1 -- start start 2024-03-25T18:25:06.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.054+0000 7f08b52d9640 1 librados: setting wanted keys 2024-03-25T18:25:06.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.054+0000 7f08b52d9640 1 librados: calling monclient init 2024-03-25T18:25:06.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.054+0000 7f08b52d9640 10 monclient: init 2024-03-25T18:25:06.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.055+0000 7f08b52d9640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:06.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.055+0000 7f08b52d9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:25:06.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.055+0000 7f08b52d9640 10 monclient(hunting): picked mon.noname-b con 0x7f08b00b4510 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:06.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.055+0000 7f08b52d9640 10 monclient(hunting): picked mon.noname-a con 0x7f08b00b4f00 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:06.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.055+0000 7f08b52d9640 10 monclient(hunting): picked mon.noname-c con 0x7f08b0142ad0 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:06.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.055+0000 7f08b52d9640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b00b10a0 con 0x7f08b00b4f00 2024-03-25T18:25:06.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.055+0000 7f08b52d9640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b00b7690 con 0x7f08b00b4510 2024-03-25T18:25:06.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.055+0000 7f08b52d9640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b00b7970 con 0x7f08b0142ad0 2024-03-25T18:25:06.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.055+0000 7f08b52d9640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:06.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.055+0000 7f08b52d9640 10 monclient(hunting): authenticate will time out at 7363.190875s 2024-03-25T18:25:06.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.055+0000 7f08aeffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f08b00b4510 0x7f08b00b86c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37724/0 (socket says 172.21.15.16:37724) 2024-03-25T18:25:06.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.055+0000 7f08aeffd640 1 -- 172.21.15.16:0/2539983572 learned_addr learned my addr 172.21.15.16:0/2539983572 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-25T18:25:06.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.056+0000 7f08977fe640 10 client.?.objecter ms_handle_connect 0x7f08b00b4510 2024-03-25T18:25:06.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.056+0000 7f08977fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:06.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.056+0000 7f08977fe640 10 client.?.objecter ms_handle_connect 0x7f08b0142ad0 2024-03-25T18:25:06.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.056+0000 7f08977fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:06.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.056+0000 7f08977fe640 10 client.?.objecter ms_handle_connect 0x7f08b00b4f00 2024-03-25T18:25:06.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.056+0000 7f08977fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:06.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.058+0000 7f08977fe640 1 -- 172.21.15.16:0/2539983572 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1482371132 0 0) 0x7f08b00b7690 con 0x7f08b00b4510 2024-03-25T18:25:06.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.058+0000 7f08977fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:06.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.058+0000 7f08977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:06.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.058+0000 7f08977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:06.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.058+0000 7f08977fe640 10 monclient(hunting): my global_id is 4114 2024-03-25T18:25:06.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.058+0000 7f08977fe640 1 -- 172.21.15.16:0/2539983572 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f089c004eb0 con 0x7f08b00b4510 2024-03-25T18:25:06.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.058+0000 7f08977fe640 1 -- 172.21.15.16:0/2539983572 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2559810498 0 0) 0x7f08b00b7970 con 0x7f08b0142ad0 2024-03-25T18:25:06.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.058+0000 7f08977fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:06.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.058+0000 7f08977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:06.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.058+0000 7f08977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.058+0000 7f08977fe640 10 monclient(hunting): my global_id is 4118 2024-03-25T18:25:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.058+0000 7f08977fe640 1 -- 172.21.15.16:0/2539983572 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08b00b7690 con 0x7f08b0142ad0 2024-03-25T18:25:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.058+0000 7f08977fe640 1 -- 172.21.15.16:0/2539983572 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2865233523 0 0) 0x7f08b00b10a0 con 0x7f08b00b4f00 2024-03-25T18:25:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.058+0000 7f08977fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.058+0000 7f08977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.058+0000 7f08977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.058+0000 7f08977fe640 10 monclient(hunting): my global_id is 4119 2024-03-25T18:25:06.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.058+0000 7f08977fe640 1 -- 172.21.15.16:0/2539983572 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08b00b7970 con 0x7f08b00b4f00 2024-03-25T18:25:06.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.063+0000 7f08977fe640 1 -- 172.21.15.16:0/2539983572 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2394311223 0 0) 0x7f089c004eb0 con 0x7f08b00b4510 2024-03-25T18:25:06.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.063+0000 7f08977fe640 1 -- 172.21.15.16:0/2539983572 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08b00b10a0 con 0x7f08b00b4510 2024-03-25T18:25:06.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.063+0000 7f08977fe640 1 -- 172.21.15.16:0/2539983572 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f08a4001c00 con 0x7f08b00b4510 2024-03-25T18:25:06.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.063+0000 7f08977fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:06.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.063+0000 7f08977fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:06.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.063+0000 7f08977fe640 10 monclient(hunting): dump: 2024-03-25T18:25:06.065 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:06.065 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:06.065 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:06.065 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:06.065 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:06.065 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:06.065 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:06.065 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:06.065 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:06.065 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:06.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.063+0000 7f08977fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:06.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.063+0000 7f08977fe640 1 -- 172.21.15.16:0/2539983572 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3666256406 0 0) 0x7f08b00b7690 con 0x7f08b0142ad0 2024-03-25T18:25:06.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.063+0000 7f08977fe640 1 -- 172.21.15.16:0/2539983572 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f089c004eb0 con 0x7f08b0142ad0 2024-03-25T18:25:06.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.063+0000 7f08977fe640 1 -- 172.21.15.16:0/2539983572 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f08a8001b50 con 0x7f08b0142ad0 2024-03-25T18:25:06.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.063+0000 7f08977fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:06.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.063+0000 7f08977fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:06.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.063+0000 7f08977fe640 10 monclient(hunting): dump: 2024-03-25T18:25:06.066 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:06.066 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:06.066 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:06.066 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:06.066 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:06.066 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:06.066 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:06.066 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:06.067 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:06.067 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:06.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.063+0000 7f08977fe640 1 -- 172.21.15.16:0/2539983572 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2537378146 0 0) 0x7f08b00b7970 con 0x7f08b00b4f00 2024-03-25T18:25:06.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.063+0000 7f08977fe640 1 -- 172.21.15.16:0/2539983572 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08b00b7690 con 0x7f08b00b4f00 2024-03-25T18:25:06.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.063+0000 7f08977fe640 1 -- 172.21.15.16:0/2539983572 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f08a00018b0 con 0x7f08b00b4f00 2024-03-25T18:25:06.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.063+0000 7f08977fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:06.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.063+0000 7f08977fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:06.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.063+0000 7f08977fe640 10 monclient(hunting): dump: 2024-03-25T18:25:06.067 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:06.067 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:06.067 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:06.067 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:06.067 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:06.067 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:06.067 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:06.068 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:06.068 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:06.068 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:06.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.067+0000 7f08977fe640 1 -- 172.21.15.16:0/2539983572 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 702237657 0 0) 0x7f08b00b7690 con 0x7f08b00b4f00 2024-03-25T18:25:06.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.067+0000 7f08977fe640 1 -- 172.21.15.16:0/2539983572 >> v1:172.21.15.3:6790/0 conn(0x7f08b0142ad0 legacy=0x7f08b01e21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:06.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.067+0000 7f08977fe640 1 -- 172.21.15.16:0/2539983572 >> v1:172.21.15.16:6789/0 conn(0x7f08b00b4510 legacy=0x7f08b00b86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:06.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.067+0000 7f08977fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:06.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.067+0000 7f08977fe640 1 monclient: found mon.a 2024-03-25T18:25:06.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.067+0000 7f08977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:06.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.067+0000 7f08977fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:06.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.067+0000 7f08977fe640 1 -- 172.21.15.16:0/2539983572 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b00b7c50 con 0x7f08b00b4f00 2024-03-25T18:25:06.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.067+0000 7f08977fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:06.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.067+0000 7f08977fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:06.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.067+0000 7f08977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:06.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.067+0000 7f08b52d9640 5 monclient: authenticate success, global_id 4119 2024-03-25T18:25:06.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.067+0000 7f08b52d9640 10 monclient: _renew_subs 2024-03-25T18:25:06.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.067+0000 7f08b52d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:06.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.067+0000 7f08b52d9640 1 -- 172.21.15.16:0/2539983572 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08b00b0d30 con 0x7f08b00b4f00 2024-03-25T18:25:06.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.067+0000 7f08b52d9640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:06.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.067+0000 7f08b52d9640 10 monclient: _renew_subs 2024-03-25T18:25:06.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.067+0000 7f08b52d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:06.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.067+0000 7f08b52d9640 1 -- 172.21.15.16:0/2539983572 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08b01d8370 con 0x7f08b00b4f00 2024-03-25T18:25:06.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.067+0000 7f08b52d9640 1 librados: init done 2024-03-25T18:25:06.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.068+0000 7f08b52d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:06.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.068+0000 7f08b52d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:06.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.068+0000 7f08b52d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:06.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.068+0000 7f08b52d9640 1 -- 172.21.15.16:0/2539983572 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08b01d86c0 con 0x7f08b00b4f00 2024-03-25T18:25:06.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.071+0000 7f08977fe640 1 -- 172.21.15.16:0/2539983572 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08a0001d00 con 0x7f08b00b4f00 2024-03-25T18:25:06.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.071+0000 7f08977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:06.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.071+0000 7f08977fe640 1 -- 172.21.15.16:0/2539983572 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f08a0003810 con 0x7f08b00b4f00 2024-03-25T18:25:06.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.071+0000 7f08977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:06.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.071+0000 7f08977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:06.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.071+0000 7f08977fe640 10 monclient: dump: 2024-03-25T18:25:06.073 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:06.073 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:06.073 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:06.073 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:06.073 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:06.073 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:06.073 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:06.073 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:06.073 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:06.074 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:06.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.072+0000 7f08977fe640 1 -- 172.21.15.16:0/2539983572 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f08a0003d40 con 0x7f08b00b4f00 2024-03-25T18:25:06.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.071+0000 7f08ad7fa640 4 set_mon_vals no callback set 2024-03-25T18:25:06.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.072+0000 7f08977fe640 1 -- 172.21.15.16:0/2539983572 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 2946279617 0 0) 0x7f08a0005300 con 0x7f08b00b4f00 2024-03-25T18:25:06.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.072+0000 7f08977fe640 10 client.4119.objecter ms_dispatch 0x7f08b0000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T18:25:06.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.072+0000 7f08977fe640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T18:25:06.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.072+0000 7f08977fe640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T18:25:06.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.072+0000 7f08977fe640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T18:25:06.102 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-25T18:25:06.101+0000 7f0c86a58040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:25:06.158 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-25T18:25:06.158+0000 7f0c86a58040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:25:06.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.173+0000 7f08977fe640 1 -- 172.21.15.16:0/2539983572 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f08a00056e0 con 0x7f08b00b4f00 2024-03-25T18:25:06.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.173+0000 7f08977fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:06.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.173+0000 7f08977fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:06.183 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-03-25T18:25:06.183+0000 7f49c1ff1040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:25:06.203 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-25T18:25:06.202+0000 7f0c86a58040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:25:06.240 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-03-25T18:25:06.239+0000 7f49c1ff1040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:25:06.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.246+0000 7f08b52d9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d8dee66f-c1ad-477e-8d93-f1bf6ec49cac", "id": 0}] 2024-03-25T18:25:06.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.246+0000 7f08b52d9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d8dee66f-c1ad-477e-8d93-f1bf6ec49cac", "id": 0}] 2024-03-25T18:25:06.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.246+0000 7f08b52d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:06.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.246+0000 7f08b52d9640 1 -- 172.21.15.16:0/2539983572 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d8dee66f-c1ad-477e-8d93-f1bf6ec49cac", "id": 0} v 0) v1 -- 0x7f08b004f730 con 0x7f08b00b4f00 2024-03-25T18:25:06.286 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-03-25T18:25:06.285+0000 7f49c1ff1040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:25:06.325 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-25T18:25:06.323+0000 7f0c86a58040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:25:06.342 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-03-25T18:25:06.341+0000 7f49c1ff1040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:25:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.356+0000 7f08977fe640 1 -- 172.21.15.16:0/2539983572 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d8dee66f-c1ad-477e-8d93-f1bf6ec49cac", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 399463766 0 732707515) 0x7f08a0005950 con 0x7f08b00b4f00 2024-03-25T18:25:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.356+0000 7f08977fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d8dee66f-c1ad-477e-8d93-f1bf6ec49cac", "id": 0}] 2024-03-25T18:25:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.356+0000 7f08977fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:06.357 INFO:teuthology.orchestra.run.smithi016.stdout:0 2024-03-25T18:25:06.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.357+0000 7f08b52d9640 10 librados: watch_flush enter 2024-03-25T18:25:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.357+0000 7f08b52d9640 10 librados: watch_flush exit 2024-03-25T18:25:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.357+0000 7f08b52d9640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T18:25:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.357+0000 7f08b52d9640 10 client.4119.objecter successfully canceled tick 2024-03-25T18:25:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.357+0000 7f08b52d9640 10 monclient: shutdown 2024-03-25T18:25:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.357+0000 7f08b52d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.357+0000 7f08b52d9640 1 -- 172.21.15.16:0/2539983572 >> v1:172.21.15.3:6789/0 conn(0x7f08b00b4f00 legacy=0x7f08b01d7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.357+0000 7f08b52d9640 1 -- 172.21.15.16:0/2539983572 shutdown_connections 2024-03-25T18:25:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.357+0000 7f08b52d9640 1 -- 172.21.15.16:0/2539983572 >> 172.21.15.16:0/2539983572 conn(0x7f08b0066b00 msgr2=0x7f08b01402c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.357+0000 7f08b52d9640 1 -- 172.21.15.16:0/2539983572 shutdown_connections 2024-03-25T18:25:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.357+0000 7f08b52d9640 1 -- 172.21.15.16:0/2539983572 wait complete. 2024-03-25T18:25:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.357+0000 7f08b52d9640 1 librados: shutdown 2024-03-25T18:25:06.369 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd new 41c16611-a0ff-4bd4-8af8-362561ba9102 1 2024-03-25T18:25:06.388 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-03-25T18:25:06.388+0000 7f49c1ff1040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:25:06.418 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-25T18:25:06.417+0000 7f0c86a58040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:25:06.466 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-25T18:25:06.465+0000 7f0c86a58040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:25:06.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.467+0000 7fce2e3cf640 10 monclient: get_monmap_and_config 2024-03-25T18:25:06.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.467+0000 7fce2e3cf640 10 monclient: build_initial_monmap 2024-03-25T18:25:06.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.467+0000 7fce2e3cf640 10 monclient: monmap: 2024-03-25T18:25:06.468 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 0 2024-03-25T18:25:06.469 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:06.469 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:25:06.468967+0000 2024-03-25T18:25:06.469 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:25:06.468967+0000 2024-03-25T18:25:06.469 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:06.469 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:06.469 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:06.469 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:25:06.469 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:25:06.469 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:06.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.468+0000 7fce2e3cf640 1 Processor -- start 2024-03-25T18:25:06.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.468+0000 7fce2e3cf640 1 -- start start 2024-03-25T18:25:06.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.468+0000 7fce2e3cf640 10 monclient: init 2024-03-25T18:25:06.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.469+0000 7fce2e3cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:06.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.469+0000 7fce2e3cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:06.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.469+0000 7fce2e3cf640 10 monclient(hunting): picked mon.noname-b con 0x7fce280b4b50 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:06.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.469+0000 7fce2e3cf640 10 monclient(hunting): picked mon.noname-c con 0x7fce2813ff20 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:06.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.469+0000 7fce2e3cf640 10 monclient(hunting): picked mon.noname-a con 0x7fce28142dd0 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:06.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.469+0000 7fce2e3cf640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce280ab440 con 0x7fce28142dd0 2024-03-25T18:25:06.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.469+0000 7fce2e3cf640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce280ab6b0 con 0x7fce2813ff20 2024-03-25T18:25:06.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.469+0000 7fce2e3cf640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce280a9f10 con 0x7fce280b4b50 2024-03-25T18:25:06.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.469+0000 7fce2e3cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:06.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.469+0000 7fce2e3cf640 10 monclient(hunting): authenticate will time out at 7363.604901s 2024-03-25T18:25:06.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.469+0000 7fce2c96c640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fce28142dd0 0x7fce2814df50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.16:45480/0 (socket says 172.21.15.16:45480) 2024-03-25T18:25:06.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.469+0000 7fce2c96c640 1 -- 172.21.15.16:0/3336720834 learned_addr learned my addr 172.21.15.16:0/3336720834 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-25T18:25:06.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.472+0000 7fce26ffd640 1 -- 172.21.15.16:0/3336720834 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851365719 0 0) 0x7fce280ab440 con 0x7fce28142dd0 2024-03-25T18:25:06.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.472+0000 7fce26ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:06.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.472+0000 7fce26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:06.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.472+0000 7fce26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:06.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.472+0000 7fce26ffd640 10 monclient(hunting): my global_id is 4122 2024-03-25T18:25:06.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.472+0000 7fce26ffd640 1 -- 172.21.15.16:0/3336720834 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce10003520 con 0x7fce28142dd0 2024-03-25T18:25:06.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.474+0000 7fce26ffd640 1 -- 172.21.15.16:0/3336720834 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723462568 0 0) 0x7fce280ab6b0 con 0x7fce2813ff20 2024-03-25T18:25:06.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.474+0000 7fce26ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:06.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.474+0000 7fce26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:06.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.474+0000 7fce26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:06.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.474+0000 7fce26ffd640 10 monclient(hunting): my global_id is 4117 2024-03-25T18:25:06.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.474+0000 7fce26ffd640 1 -- 172.21.15.16:0/3336720834 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce280ab440 con 0x7fce2813ff20 2024-03-25T18:25:06.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.474+0000 7fce26ffd640 1 -- 172.21.15.16:0/3336720834 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734193163 0 0) 0x7fce280a9f10 con 0x7fce280b4b50 2024-03-25T18:25:06.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.474+0000 7fce26ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:06.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.474+0000 7fce26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:06.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.474+0000 7fce26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:06.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.474+0000 7fce26ffd640 10 monclient(hunting): my global_id is 4121 2024-03-25T18:25:06.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.474+0000 7fce26ffd640 1 -- 172.21.15.16:0/3336720834 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce280ab6b0 con 0x7fce280b4b50 2024-03-25T18:25:06.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.475+0000 7fce26ffd640 1 -- 172.21.15.16:0/3336720834 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2923314940 0 0) 0x7fce10003520 con 0x7fce28142dd0 2024-03-25T18:25:06.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.475+0000 7fce26ffd640 1 -- 172.21.15.16:0/3336720834 >> v1:172.21.15.3:6790/0 conn(0x7fce280b4b50 legacy=0x7fce2813f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:06.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.475+0000 7fce26ffd640 1 -- 172.21.15.16:0/3336720834 >> v1:172.21.15.16:6789/0 conn(0x7fce2813ff20 legacy=0x7fce281426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:06.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.475+0000 7fce26ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:06.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.475+0000 7fce26ffd640 1 monclient: found mon.noname-a 2024-03-25T18:25:06.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.475+0000 7fce26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:06.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.475+0000 7fce26ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:25:06.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.475+0000 7fce26ffd640 1 -- 172.21.15.16:0/3336720834 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce280aa0a0 con 0x7fce28142dd0 2024-03-25T18:25:06.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.475+0000 7fce26ffd640 10 monclient: _finish_auth 0 2024-03-25T18:25:06.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.475+0000 7fce26ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:06.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.475+0000 7fce26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:06.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.475+0000 7fce2e3cf640 5 monclient: authenticate success, global_id 4122 2024-03-25T18:25:06.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.475+0000 7fce2e3cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:06.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.475+0000 7fce26ffd640 1 -- 172.21.15.16:0/3336720834 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fce200019d0 con 0x7fce28142dd0 2024-03-25T18:25:06.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.475+0000 7fce26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:06.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.475+0000 7fce26ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:06.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.475+0000 7fce26ffd640 10 monclient: dump: 2024-03-25T18:25:06.477 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:06.478 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:06.478 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:06.478 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:06.478 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:06.478 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:06.478 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:06.478 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:06.478 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:06.478 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:06.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.475+0000 7fce2e3cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:06.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.484+0000 7fce26ffd640 1 -- 172.21.15.16:0/3336720834 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce20001bb0 con 0x7fce28142dd0 2024-03-25T18:25:06.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.484+0000 7fce26ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:06.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.484+0000 7fce26ffd640 1 -- 172.21.15.16:0/3336720834 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fce20001f40 con 0x7fce28142dd0 2024-03-25T18:25:06.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.484+0000 7fce26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:06.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.484+0000 7fce26ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:06.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.484+0000 7fce26ffd640 10 monclient: dump: 2024-03-25T18:25:06.485 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:06.486 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:06.486 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:06.486 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:06.486 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:06.486 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:06.486 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:06.486 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:06.486 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:06.486 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:06.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.484+0000 7fce2e3cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:06.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.484+0000 7fce2e3cf640 4 set_mon_vals no callback set 2024-03-25T18:25:06.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.485+0000 7fce2e3cf640 10 monclient: shutdown 2024-03-25T18:25:06.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.485+0000 7fce2e3cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:06.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.485+0000 7fce2e3cf640 1 -- 172.21.15.16:0/3336720834 >> v1:172.21.15.3:6789/0 conn(0x7fce28142dd0 legacy=0x7fce2814df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:06.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.485+0000 7fce2e3cf640 1 -- 172.21.15.16:0/3336720834 shutdown_connections 2024-03-25T18:25:06.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.485+0000 7fce2e3cf640 1 -- 172.21.15.16:0/3336720834 >> 172.21.15.16:0/3336720834 conn(0x7fce28066920 msgr2=0x7fce28066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:06.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.485+0000 7fce2e3cf640 1 -- 172.21.15.16:0/3336720834 shutdown_connections 2024-03-25T18:25:06.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.485+0000 7fce2e3cf640 1 -- 172.21.15.16:0/3336720834 wait complete. 2024-03-25T18:25:06.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.486+0000 7fce2e3cf640 10 monclient: build_initial_monmap 2024-03-25T18:25:06.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.486+0000 7fce2e3cf640 10 monclient: monmap: 2024-03-25T18:25:06.489 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 0 2024-03-25T18:25:06.489 INFO:teuthology.orchestra.run.smithi016.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:06.489 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 0.000000 2024-03-25T18:25:06.489 INFO:teuthology.orchestra.run.smithi016.stderr:created 0.000000 2024-03-25T18:25:06.489 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:06.489 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 1 2024-03-25T18:25:06.489 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:06.489 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:25:06.489 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:25:06.489 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:06.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.486+0000 7fce2e3cf640 1 librados: starting msgr at 2024-03-25T18:25:06.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.486+0000 7fce2e3cf640 1 librados: starting objecter 2024-03-25T18:25:06.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.486+0000 7fce2e3cf640 1 Processor -- start 2024-03-25T18:25:06.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.487+0000 7fce2e3cf640 1 -- start start 2024-03-25T18:25:06.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.487+0000 7fce2e3cf640 1 librados: setting wanted keys 2024-03-25T18:25:06.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.487+0000 7fce2e3cf640 1 librados: calling monclient init 2024-03-25T18:25:06.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.487+0000 7fce2e3cf640 10 monclient: init 2024-03-25T18:25:06.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.487+0000 7fce2e3cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:06.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.487+0000 7fce2e3cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:06.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.487+0000 7fce2e3cf640 10 monclient(hunting): picked mon.noname-c con 0x7fce280b4b50 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:06.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.488+0000 7fce2e3cf640 10 monclient(hunting): picked mon.noname-a con 0x7fce2813ff20 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:06.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.488+0000 7fce2e3cf640 10 monclient(hunting): picked mon.noname-b con 0x7fce28142dd0 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:06.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.488+0000 7fce2e3cf640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce28144650 con 0x7fce2813ff20 2024-03-25T18:25:06.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.488+0000 7fce2e3cf640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce281448c0 con 0x7fce28142dd0 2024-03-25T18:25:06.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.488+0000 7fce2e3cf640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce28142120 con 0x7fce280b4b50 2024-03-25T18:25:06.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.488+0000 7fce2e3cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:06.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.488+0000 7fce2e3cf640 10 monclient(hunting): authenticate will time out at 7363.623647s 2024-03-25T18:25:06.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.488+0000 7fce277fe640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fce2813ff20 0x7fce281455c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.16:45488/0 (socket says 172.21.15.16:45488) 2024-03-25T18:25:06.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.488+0000 7fce277fe640 1 -- 172.21.15.16:0/1106748978 learned_addr learned my addr 172.21.15.16:0/1106748978 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-25T18:25:06.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.488+0000 7fce24ff9640 10 client.?.objecter ms_handle_connect 0x7fce2813ff20 2024-03-25T18:25:06.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.488+0000 7fce24ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:06.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.491+0000 7fce24ff9640 1 -- 172.21.15.16:0/1106748978 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1122975215 0 0) 0x7fce28144650 con 0x7fce2813ff20 2024-03-25T18:25:06.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.491+0000 7fce24ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:06.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.491+0000 7fce24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:06.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.491+0000 7fce24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:06.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.491+0000 7fce24ff9640 10 monclient(hunting): my global_id is 4125 2024-03-25T18:25:06.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.491+0000 7fce24ff9640 1 -- 172.21.15.16:0/1106748978 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce14004eb0 con 0x7fce2813ff20 2024-03-25T18:25:06.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.491+0000 7fce24ff9640 10 client.?.objecter ms_handle_connect 0x7fce280b4b50 2024-03-25T18:25:06.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.491+0000 7fce24ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:06.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.492+0000 7fce24ff9640 10 client.?.objecter ms_handle_connect 0x7fce28142dd0 2024-03-25T18:25:06.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.492+0000 7fce24ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:06.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.496+0000 7fce24ff9640 1 -- 172.21.15.16:0/1106748978 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4058570181 0 0) 0x7fce14004eb0 con 0x7fce2813ff20 2024-03-25T18:25:06.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.496+0000 7fce24ff9640 1 -- 172.21.15.16:0/1106748978 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce28144650 con 0x7fce2813ff20 2024-03-25T18:25:06.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.496+0000 7fce24ff9640 1 -- 172.21.15.16:0/1106748978 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fce1c0019b0 con 0x7fce2813ff20 2024-03-25T18:25:06.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.496+0000 7fce24ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:06.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.496+0000 7fce24ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:06.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.496+0000 7fce24ff9640 10 monclient(hunting): dump: 2024-03-25T18:25:06.497 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:06.498 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:06.498 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:06.498 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:06.498 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:06.498 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:06.498 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:06.498 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:06.498 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:06.498 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:06.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.496+0000 7fce24ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:06.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.509+0000 7fce24ff9640 1 -- 172.21.15.16:0/1106748978 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4211070001 0 0) 0x7fce28144650 con 0x7fce2813ff20 2024-03-25T18:25:06.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.509+0000 7fce24ff9640 1 -- 172.21.15.16:0/1106748978 >> v1:172.21.15.3:6790/0 conn(0x7fce280b4b50 legacy=0x7fce28130f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:06.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.509+0000 7fce24ff9640 1 -- 172.21.15.16:0/1106748978 >> v1:172.21.15.16:6789/0 conn(0x7fce28142dd0 legacy=0x7fce28145cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:06.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.509+0000 7fce24ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:25:06.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.509+0000 7fce24ff9640 1 monclient: found mon.a 2024-03-25T18:25:06.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.509+0000 7fce24ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:06.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.509+0000 7fce24ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:06.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.509+0000 7fce24ff9640 1 -- 172.21.15.16:0/1106748978 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce28142400 con 0x7fce2813ff20 2024-03-25T18:25:06.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.509+0000 7fce24ff9640 10 monclient: _finish_auth 0 2024-03-25T18:25:06.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.509+0000 7fce24ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:25:06.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.509+0000 7fce24ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:06.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.509+0000 7fce2e3cf640 5 monclient: authenticate success, global_id 4125 2024-03-25T18:25:06.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.509+0000 7fce2e3cf640 10 monclient: _renew_subs 2024-03-25T18:25:06.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.509+0000 7fce2e3cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:06.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.509+0000 7fce2e3cf640 1 -- 172.21.15.16:0/1106748978 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce28145030 con 0x7fce2813ff20 2024-03-25T18:25:06.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.509+0000 7fce2e3cf640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:06.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.509+0000 7fce2e3cf640 10 monclient: _renew_subs 2024-03-25T18:25:06.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.509+0000 7fce2e3cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:06.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.509+0000 7fce2e3cf640 1 -- 172.21.15.16:0/1106748978 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce28143b90 con 0x7fce2813ff20 2024-03-25T18:25:06.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.509+0000 7fce2e3cf640 1 librados: init done 2024-03-25T18:25:06.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.510+0000 7fce2e3cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:06.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.510+0000 7fce2e3cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:06.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.510+0000 7fce2e3cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:06.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.510+0000 7fce2e3cf640 1 -- 172.21.15.16:0/1106748978 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce28145220 con 0x7fce2813ff20 2024-03-25T18:25:06.513 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-03-25T18:25:06.512+0000 7f49c1ff1040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:25:06.525 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-25T18:25:06.524+0000 7f0c86a58040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:25:06.576 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-25T18:25:06.575+0000 7f0c86a58040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:25:06.586 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-03-25T18:25:06.585+0000 7f49c1ff1040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:25:06.633 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-03-25T18:25:06.632+0000 7f49c1ff1040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:25:06.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.653+0000 7fce24ff9640 1 -- 172.21.15.16:0/1106748978 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce1c001c70 con 0x7fce2813ff20 2024-03-25T18:25:06.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.653+0000 7fce24ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:06.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.653+0000 7fce24ff9640 1 -- 172.21.15.16:0/1106748978 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fce1c003a60 con 0x7fce2813ff20 2024-03-25T18:25:06.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.653+0000 7fce24ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:06.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.653+0000 7fce24ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:06.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.653+0000 7fce24ff9640 10 monclient: dump: 2024-03-25T18:25:06.654 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:06.654 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:06.654 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:06.654 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:06.655 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:06.655 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:06.655 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:06.655 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:06.655 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:06.655 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:06.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.653+0000 7fce24ff9640 1 -- 172.21.15.16:0/1106748978 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fce1c004010 con 0x7fce2813ff20 2024-03-25T18:25:06.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.653+0000 7fce267fc640 4 set_mon_vals no callback set 2024-03-25T18:25:06.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.654+0000 7fce24ff9640 1 -- 172.21.15.16:0/1106748978 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 4229794011 0 0) 0x7fce1c006620 con 0x7fce2813ff20 2024-03-25T18:25:06.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.654+0000 7fce24ff9640 10 client.4125.objecter ms_dispatch 0x7fce28000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T18:25:06.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.654+0000 7fce24ff9640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T18:25:06.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.654+0000 7fce24ff9640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-03-25T18:25:06.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.654+0000 7fce24ff9640 20 client.4125.objecter dump_active .. 0 homeless 2024-03-25T18:25:06.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.654+0000 7fce24ff9640 1 -- 172.21.15.16:0/1106748978 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fce1c006970 con 0x7fce2813ff20 2024-03-25T18:25:06.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.654+0000 7fce24ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:06.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.654+0000 7fce24ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:06.688 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-03-25T18:25:06.687+0000 7f49c1ff1040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:25:06.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.728+0000 7fce2e3cf640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "41c16611-a0ff-4bd4-8af8-362561ba9102", "id": 1}] 2024-03-25T18:25:06.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.728+0000 7fce2e3cf640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "41c16611-a0ff-4bd4-8af8-362561ba9102", "id": 1}] 2024-03-25T18:25:06.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.728+0000 7fce2e3cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:06.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.728+0000 7fce2e3cf640 1 -- 172.21.15.16:0/1106748978 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd new", "uuid": "41c16611-a0ff-4bd4-8af8-362561ba9102", "id": 1} v 0) v1 -- 0x7fce2804f730 con 0x7fce2813ff20 2024-03-25T18:25:06.737 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-03-25T18:25:06.736+0000 7f49c1ff1040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:25:06.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.799+0000 7fce24ff9640 1 -- 172.21.15.16:0/1106748978 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "41c16611-a0ff-4bd4-8af8-362561ba9102", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 2659098347 0 940483276) 0x7fce1c004450 con 0x7fce2813ff20 2024-03-25T18:25:06.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.799+0000 7fce24ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "41c16611-a0ff-4bd4-8af8-362561ba9102", "id": 1}] 2024-03-25T18:25:06.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.799+0000 7fce24ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:06.801 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-03-25T18:25:06.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.800+0000 7fce2e3cf640 10 librados: watch_flush enter 2024-03-25T18:25:06.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.800+0000 7fce2e3cf640 10 librados: watch_flush exit 2024-03-25T18:25:06.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.800+0000 7fce2e3cf640 20 client.4125.objecter shutdown clearing up homeless session... 2024-03-25T18:25:06.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.800+0000 7fce2e3cf640 10 client.4125.objecter successfully canceled tick 2024-03-25T18:25:06.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.800+0000 7fce2e3cf640 10 monclient: shutdown 2024-03-25T18:25:06.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.800+0000 7fce2e3cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:06.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.800+0000 7fce2e3cf640 1 -- 172.21.15.16:0/1106748978 >> v1:172.21.15.3:6789/0 conn(0x7fce2813ff20 legacy=0x7fce281455c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:06.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.800+0000 7fce2e3cf640 1 -- 172.21.15.16:0/1106748978 shutdown_connections 2024-03-25T18:25:06.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.800+0000 7fce2e3cf640 1 -- 172.21.15.16:0/1106748978 >> 172.21.15.16:0/1106748978 conn(0x7fce28066920 msgr2=0x7fce280b2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:06.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.800+0000 7fce2e3cf640 1 -- 172.21.15.16:0/1106748978 shutdown_connections 2024-03-25T18:25:06.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.800+0000 7fce2e3cf640 1 -- 172.21.15.16:0/1106748978 wait complete. 2024-03-25T18:25:06.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.800+0000 7fce2e3cf640 1 librados: shutdown 2024-03-25T18:25:06.811 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd new 3e3857a4-8eff-44cf-b75c-dacf17b4248b 2 2024-03-25T18:25:06.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.951+0000 7fcb0f406640 10 monclient: get_monmap_and_config 2024-03-25T18:25:06.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.951+0000 7fcb0f406640 10 monclient: build_initial_monmap 2024-03-25T18:25:06.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.951+0000 7fcb0f406640 10 monclient: monmap: 2024-03-25T18:25:06.953 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 0 2024-03-25T18:25:06.953 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:06.953 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:25:06.952667+0000 2024-03-25T18:25:06.953 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:25:06.952667+0000 2024-03-25T18:25:06.953 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:06.953 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:06.953 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:06.953 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:25:06.953 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:25:06.953 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:06.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.952+0000 7fcb0f406640 1 Processor -- start 2024-03-25T18:25:06.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.952+0000 7fcb0f406640 1 -- start start 2024-03-25T18:25:06.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.952+0000 7fcb0f406640 10 monclient: init 2024-03-25T18:25:06.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.952+0000 7fcb0f406640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:06.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.952+0000 7fcb0f406640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:25:06.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.952+0000 7fcb0f406640 10 monclient(hunting): picked mon.noname-c con 0x7fcb080b4b50 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:06.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.952+0000 7fcb0f406640 10 monclient(hunting): picked mon.noname-a con 0x7fcb0813ff20 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:06.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.952+0000 7fcb0f406640 10 monclient(hunting): picked mon.noname-b con 0x7fcb08142dd0 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:06.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.952+0000 7fcb0f406640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb080ab440 con 0x7fcb0813ff20 2024-03-25T18:25:06.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.952+0000 7fcb0f406640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb080ab6b0 con 0x7fcb080b4b50 2024-03-25T18:25:06.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.952+0000 7fcb0f406640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb080a9f10 con 0x7fcb08142dd0 2024-03-25T18:25:06.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.952+0000 7fcb0f406640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:06.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.952+0000 7fcb0f406640 10 monclient(hunting): authenticate will time out at 7364.088541s 2024-03-25T18:25:06.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.953+0000 7fcb0c9a1640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fcb0813ff20 0x7fcb081426c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.16:45494/0 (socket says 172.21.15.16:45494) 2024-03-25T18:25:06.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.953+0000 7fcb0c9a1640 1 -- 172.21.15.16:0/3080116 learned_addr learned my addr 172.21.15.16:0/3080116 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-25T18:25:06.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.956+0000 7fcafffff640 1 -- 172.21.15.16:0/3080116 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 542013752 0 0) 0x7fcb080ab440 con 0x7fcb0813ff20 2024-03-25T18:25:06.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.956+0000 7fcafffff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:06.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.956+0000 7fcafffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:06.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.956+0000 7fcafffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:06.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.956+0000 7fcafffff640 10 monclient(hunting): my global_id is 4128 2024-03-25T18:25:06.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.956+0000 7fcafffff640 1 -- 172.21.15.16:0/3080116 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaf4003520 con 0x7fcb0813ff20 2024-03-25T18:25:06.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.956+0000 7fcafffff640 1 -- 172.21.15.16:0/3080116 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858843714 0 0) 0x7fcb080ab6b0 con 0x7fcb080b4b50 2024-03-25T18:25:06.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.956+0000 7fcafffff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:06.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.956+0000 7fcafffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:06.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.956+0000 7fcafffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:06.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.956+0000 7fcafffff640 10 monclient(hunting): my global_id is 4120 2024-03-25T18:25:06.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.956+0000 7fcafffff640 1 -- 172.21.15.16:0/3080116 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb080ab440 con 0x7fcb080b4b50 2024-03-25T18:25:06.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.957+0000 7fcafffff640 1 -- 172.21.15.16:0/3080116 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 529126949 0 0) 0x7fcb080a9f10 con 0x7fcb08142dd0 2024-03-25T18:25:06.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.957+0000 7fcafffff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:06.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.957+0000 7fcafffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:06.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.957+0000 7fcafffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:06.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.957+0000 7fcafffff640 10 monclient(hunting): my global_id is 4124 2024-03-25T18:25:06.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.957+0000 7fcafffff640 1 -- 172.21.15.16:0/3080116 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb080ab6b0 con 0x7fcb08142dd0 2024-03-25T18:25:06.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.959+0000 7fcafffff640 1 -- 172.21.15.16:0/3080116 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 768891011 0 0) 0x7fcaf4003520 con 0x7fcb0813ff20 2024-03-25T18:25:06.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.959+0000 7fcafffff640 1 -- 172.21.15.16:0/3080116 >> v1:172.21.15.3:6790/0 conn(0x7fcb08142dd0 legacy=0x7fcb0814df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:06.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.959+0000 7fcafffff640 1 -- 172.21.15.16:0/3080116 >> v1:172.21.15.16:6789/0 conn(0x7fcb080b4b50 legacy=0x7fcb0813f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:06.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.959+0000 7fcafffff640 10 monclient: _finish_hunting 0 2024-03-25T18:25:06.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.959+0000 7fcafffff640 1 monclient: found mon.noname-a 2024-03-25T18:25:06.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.959+0000 7fcafffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:06.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.960+0000 7fcafffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:25:06.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.960+0000 7fcafffff640 1 -- 172.21.15.16:0/3080116 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb080aa0a0 con 0x7fcb0813ff20 2024-03-25T18:25:06.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.960+0000 7fcafffff640 10 monclient: _finish_auth 0 2024-03-25T18:25:06.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.960+0000 7fcafffff640 10 monclient: _check_auth_tickets 2024-03-25T18:25:06.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.960+0000 7fcafffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:06.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.960+0000 7fcafffff640 1 -- 172.21.15.16:0/3080116 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fcaf8001690 con 0x7fcb0813ff20 2024-03-25T18:25:06.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.960+0000 7fcafffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:06.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.960+0000 7fcafffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:06.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.960+0000 7fcafffff640 10 monclient: dump: 2024-03-25T18:25:06.962 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:06.962 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:06.962 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:06.962 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:06.962 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:06.962 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:06.962 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:06.962 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:06.962 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:06.962 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:06.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.960+0000 7fcb0f406640 5 monclient: authenticate success, global_id 4128 2024-03-25T18:25:06.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.960+0000 7fcb0f406640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:06.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.961+0000 7fcafffff640 1 -- 172.21.15.16:0/3080116 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcaf80017f0 con 0x7fcb0813ff20 2024-03-25T18:25:06.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.961+0000 7fcafffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:06.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.961+0000 7fcafffff640 1 -- 172.21.15.16:0/3080116 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fcaf8001b80 con 0x7fcb0813ff20 2024-03-25T18:25:06.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.961+0000 7fcafffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:06.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.961+0000 7fcafffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:06.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.961+0000 7fcafffff640 10 monclient: dump: 2024-03-25T18:25:06.966 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:06.966 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:06.966 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:06.966 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:06.966 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:06.966 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:06.967 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:06.967 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:06.967 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:06.967 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:06.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.961+0000 7fcb0f406640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:06.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.961+0000 7fcb0f406640 4 set_mon_vals no callback set 2024-03-25T18:25:06.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.962+0000 7fcb0f406640 10 monclient: shutdown 2024-03-25T18:25:06.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.962+0000 7fcb0f406640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:06.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.962+0000 7fcb0f406640 1 -- 172.21.15.16:0/3080116 >> v1:172.21.15.3:6789/0 conn(0x7fcb0813ff20 legacy=0x7fcb081426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:06.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.962+0000 7fcb0f406640 1 -- 172.21.15.16:0/3080116 shutdown_connections 2024-03-25T18:25:06.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.962+0000 7fcb0f406640 1 -- 172.21.15.16:0/3080116 >> 172.21.15.16:0/3080116 conn(0x7fcb08066920 msgr2=0x7fcb08066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:06.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.962+0000 7fcb0f406640 1 -- 172.21.15.16:0/3080116 shutdown_connections 2024-03-25T18:25:06.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.962+0000 7fcb0f406640 1 -- 172.21.15.16:0/3080116 wait complete. 2024-03-25T18:25:06.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.963+0000 7fcb0f406640 10 monclient: build_initial_monmap 2024-03-25T18:25:06.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.963+0000 7fcb0f406640 10 monclient: monmap: 2024-03-25T18:25:06.968 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 0 2024-03-25T18:25:06.968 INFO:teuthology.orchestra.run.smithi016.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:06.968 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 0.000000 2024-03-25T18:25:06.968 INFO:teuthology.orchestra.run.smithi016.stderr:created 0.000000 2024-03-25T18:25:06.968 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:06.968 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 1 2024-03-25T18:25:06.968 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:06.968 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:25:06.968 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:25:06.968 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:06.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.963+0000 7fcb0f406640 1 librados: starting msgr at 2024-03-25T18:25:06.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.963+0000 7fcb0f406640 1 librados: starting objecter 2024-03-25T18:25:06.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.963+0000 7fcb0f406640 1 Processor -- start 2024-03-25T18:25:06.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.963+0000 7fcb0f406640 1 -- start start 2024-03-25T18:25:06.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.963+0000 7fcb0f406640 1 librados: setting wanted keys 2024-03-25T18:25:06.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.963+0000 7fcb0f406640 1 librados: calling monclient init 2024-03-25T18:25:06.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.963+0000 7fcb0f406640 10 monclient: init 2024-03-25T18:25:06.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.964+0000 7fcb0f406640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:06.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.964+0000 7fcb0f406640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:06.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.964+0000 7fcb0f406640 10 monclient(hunting): picked mon.noname-c con 0x7fcb080b4b50 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:06.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.964+0000 7fcb0f406640 10 monclient(hunting): picked mon.noname-a con 0x7fcb0813ff20 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:06.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.964+0000 7fcb0f406640 10 monclient(hunting): picked mon.noname-b con 0x7fcb08142dd0 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:06.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.964+0000 7fcb0f406640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb08134c40 con 0x7fcb0813ff20 2024-03-25T18:25:06.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.964+0000 7fcb0f406640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb0813a460 con 0x7fcb08142dd0 2024-03-25T18:25:06.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.964+0000 7fcb0f406640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb08144780 con 0x7fcb080b4b50 2024-03-25T18:25:06.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.964+0000 7fcb0f406640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:06.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.964+0000 7fcb0f406640 10 monclient(hunting): authenticate will time out at 7364.099823s 2024-03-25T18:25:06.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.964+0000 7fcb0c9a1640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fcb0813ff20 0x7fcb08133e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.16:45504/0 (socket says 172.21.15.16:45504) 2024-03-25T18:25:06.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.964+0000 7fcb0c9a1640 1 -- 172.21.15.16:0/568119960 learned_addr learned my addr 172.21.15.16:0/568119960 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-25T18:25:06.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.964+0000 7fcafdffb640 10 client.?.objecter ms_handle_connect 0x7fcb0813ff20 2024-03-25T18:25:06.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.964+0000 7fcafdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:06.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.965+0000 7fcafdffb640 10 client.?.objecter ms_handle_connect 0x7fcb08142dd0 2024-03-25T18:25:06.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.965+0000 7fcafdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:06.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.965+0000 7fcafdffb640 10 client.?.objecter ms_handle_connect 0x7fcb080b4b50 2024-03-25T18:25:06.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.965+0000 7fcafdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:06.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.967+0000 7fcafdffb640 1 -- 172.21.15.16:0/568119960 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286200253 0 0) 0x7fcb08134c40 con 0x7fcb0813ff20 2024-03-25T18:25:06.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.967+0000 7fcafdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:06.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.967+0000 7fcafdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:06.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.967+0000 7fcafdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:06.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.967+0000 7fcafdffb640 10 monclient(hunting): my global_id is 4131 2024-03-25T18:25:06.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.967+0000 7fcafdffb640 1 -- 172.21.15.16:0/568119960 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaec004eb0 con 0x7fcb0813ff20 2024-03-25T18:25:06.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.967+0000 7fcafdffb640 1 -- 172.21.15.16:0/568119960 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022948620 0 0) 0x7fcb0813a460 con 0x7fcb08142dd0 2024-03-25T18:25:06.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.967+0000 7fcafdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:06.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.967+0000 7fcafdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:06.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.967+0000 7fcafdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:06.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.967+0000 7fcafdffb640 10 monclient(hunting): my global_id is 4123 2024-03-25T18:25:06.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.967+0000 7fcafdffb640 1 -- 172.21.15.16:0/568119960 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb08134c40 con 0x7fcb08142dd0 2024-03-25T18:25:06.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.967+0000 7fcafdffb640 1 -- 172.21.15.16:0/568119960 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 6441982 0 0) 0x7fcb08144780 con 0x7fcb080b4b50 2024-03-25T18:25:06.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.967+0000 7fcafdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:06.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.967+0000 7fcafdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:06.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.967+0000 7fcafdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:06.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.967+0000 7fcafdffb640 10 monclient(hunting): my global_id is 4127 2024-03-25T18:25:06.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.967+0000 7fcafdffb640 1 -- 172.21.15.16:0/568119960 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb0813a460 con 0x7fcb080b4b50 2024-03-25T18:25:06.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.971+0000 7fcafdffb640 1 -- 172.21.15.16:0/568119960 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2339092484 0 0) 0x7fcb08134c40 con 0x7fcb08142dd0 2024-03-25T18:25:06.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.971+0000 7fcafdffb640 1 -- 172.21.15.16:0/568119960 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb08144780 con 0x7fcb08142dd0 2024-03-25T18:25:06.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.971+0000 7fcafdffb640 1 -- 172.21.15.16:0/568119960 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fcb00001b50 con 0x7fcb08142dd0 2024-03-25T18:25:06.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.971+0000 7fcafdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:06.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.971+0000 7fcafdffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:06.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.971+0000 7fcafdffb640 10 monclient(hunting): dump: 2024-03-25T18:25:06.974 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:06.974 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:06.974 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:06.974 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:06.974 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:06.974 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:06.974 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:06.975 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:06.975 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:06.975 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:06.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.971+0000 7fcafdffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:06.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.972+0000 7fcafdffb640 1 -- 172.21.15.16:0/568119960 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 606480386 0 0) 0x7fcaec004eb0 con 0x7fcb0813ff20 2024-03-25T18:25:06.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.973+0000 7fcafdffb640 1 -- 172.21.15.16:0/568119960 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb08134c40 con 0x7fcb0813ff20 2024-03-25T18:25:06.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.973+0000 7fcafdffb640 1 -- 172.21.15.16:0/568119960 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fcaf8001b00 con 0x7fcb0813ff20 2024-03-25T18:25:06.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.973+0000 7fcafdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:06.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.973+0000 7fcafdffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:06.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.973+0000 7fcafdffb640 10 monclient(hunting): dump: 2024-03-25T18:25:06.975 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:06.975 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:06.975 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:06.975 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:06.976 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:06.976 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:06.976 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:06.976 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:06.976 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:06.976 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:06.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.974+0000 7fcafdffb640 1 -- 172.21.15.16:0/568119960 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2692512496 0 0) 0x7fcb0813a460 con 0x7fcb080b4b50 2024-03-25T18:25:06.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.974+0000 7fcafdffb640 1 -- 172.21.15.16:0/568119960 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcaec004eb0 con 0x7fcb080b4b50 2024-03-25T18:25:06.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.974+0000 7fcafdffb640 1 -- 172.21.15.16:0/568119960 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fcb040018f0 con 0x7fcb080b4b50 2024-03-25T18:25:06.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.974+0000 7fcafdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:06.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.974+0000 7fcafdffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:06.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.974+0000 7fcafdffb640 10 monclient(hunting): dump: 2024-03-25T18:25:06.978 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:06.978 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:06.978 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:06.978 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:06.978 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:06.978 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:06.978 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:06.978 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:06.978 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:06.978 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:06.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.976+0000 7fcafdffb640 1 -- 172.21.15.16:0/568119960 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3764109676 0 0) 0x7fcb08134c40 con 0x7fcb0813ff20 2024-03-25T18:25:06.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.976+0000 7fcafdffb640 1 -- 172.21.15.16:0/568119960 >> v1:172.21.15.3:6790/0 conn(0x7fcb080b4b50 legacy=0x7fcb0813a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:06.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.976+0000 7fcafdffb640 1 -- 172.21.15.16:0/568119960 >> v1:172.21.15.16:6789/0 conn(0x7fcb08142dd0 legacy=0x7fcb08134530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:06.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.976+0000 7fcafdffb640 10 monclient: _finish_hunting 0 2024-03-25T18:25:06.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.976+0000 7fcafdffb640 1 monclient: found mon.a 2024-03-25T18:25:06.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.976+0000 7fcafdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:06.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.976+0000 7fcafdffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:06.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.976+0000 7fcafdffb640 1 -- 172.21.15.16:0/568119960 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb08144a60 con 0x7fcb0813ff20 2024-03-25T18:25:06.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.976+0000 7fcafdffb640 10 monclient: _finish_auth 0 2024-03-25T18:25:06.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.976+0000 7fcafdffb640 10 monclient: _check_auth_tickets 2024-03-25T18:25:06.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.976+0000 7fcafdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:06.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.976+0000 7fcb0f406640 5 monclient: authenticate success, global_id 4131 2024-03-25T18:25:06.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.976+0000 7fcb0f406640 10 monclient: _renew_subs 2024-03-25T18:25:06.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.976+0000 7fcb0f406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:06.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.976+0000 7fcb0f406640 1 -- 172.21.15.16:0/568119960 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb08141d90 con 0x7fcb0813ff20 2024-03-25T18:25:06.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.976+0000 7fcb0f406640 10 client.4131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:06.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.976+0000 7fcb0f406640 10 monclient: _renew_subs 2024-03-25T18:25:06.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.976+0000 7fcb0f406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:06.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.976+0000 7fcb0f406640 1 -- 172.21.15.16:0/568119960 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb081daf70 con 0x7fcb0813ff20 2024-03-25T18:25:06.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.976+0000 7fcb0f406640 1 librados: init done 2024-03-25T18:25:06.980 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-25T18:25:06.978+0000 7f0c86a58040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:25:06.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.977+0000 7fcb0f406640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:06.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.977+0000 7fcb0f406640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:06.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.977+0000 7fcb0f406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:06.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:06.977+0000 7fcb0f406640 1 -- 172.21.15.16:0/568119960 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb0804f730 con 0x7fcb0813ff20 2024-03-25T18:25:07.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.078+0000 7fcafdffb640 1 -- 172.21.15.16:0/568119960 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcaf8004a90 con 0x7fcb0813ff20 2024-03-25T18:25:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.078+0000 7fcafdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.078+0000 7fcafdffb640 1 -- 172.21.15.16:0/568119960 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fcaf8004da0 con 0x7fcb0813ff20 2024-03-25T18:25:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.078+0000 7fcafdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.078+0000 7fcafdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.078+0000 7fcafdffb640 10 monclient: dump: 2024-03-25T18:25:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:07.081 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.078+0000 7fcafdffb640 1 -- 172.21.15.16:0/568119960 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fcaf8005290 con 0x7fcb0813ff20 2024-03-25T18:25:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.078+0000 7fcaff7fe640 4 set_mon_vals no callback set 2024-03-25T18:25:07.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.079+0000 7fcafdffb640 1 -- 172.21.15.16:0/568119960 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 2715799162 0 0) 0x7fcaf80066d0 con 0x7fcb0813ff20 2024-03-25T18:25:07.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.079+0000 7fcafdffb640 10 client.4131.objecter ms_dispatch 0x7fcb08000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T18:25:07.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.079+0000 7fcafdffb640 3 client.4131.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T18:25:07.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.079+0000 7fcafdffb640 3 client.4131.objecter handle_osd_map decoding full epoch 4 2024-03-25T18:25:07.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.079+0000 7fcafdffb640 20 client.4131.objecter dump_active .. 0 homeless 2024-03-25T18:25:07.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.079+0000 7fcafdffb640 1 -- 172.21.15.16:0/568119960 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fcaf80062b0 con 0x7fcb0813ff20 2024-03-25T18:25:07.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.079+0000 7fcafdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:07.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.079+0000 7fcafdffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:07.120 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-25T18:25:07.119+0000 7f0c86a58040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:25:07.142 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-03-25T18:25:07.139+0000 7f49c1ff1040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:25:07.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.153+0000 7fcb0f406640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3e3857a4-8eff-44cf-b75c-dacf17b4248b", "id": 2}] 2024-03-25T18:25:07.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.153+0000 7fcb0f406640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3e3857a4-8eff-44cf-b75c-dacf17b4248b", "id": 2}] 2024-03-25T18:25:07.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.153+0000 7fcb0f406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:07.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.153+0000 7fcb0f406640 1 -- 172.21.15.16:0/568119960 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd new", "uuid": "3e3857a4-8eff-44cf-b75c-dacf17b4248b", "id": 2} v 0) v1 -- 0x7fcb080681c0 con 0x7fcb0813ff20 2024-03-25T18:25:07.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.203+0000 7fcafdffb640 1 -- 172.21.15.16:0/568119960 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3e3857a4-8eff-44cf-b75c-dacf17b4248b", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 4036724943 0 216599125) 0x7fcaf80069c0 con 0x7fcb0813ff20 2024-03-25T18:25:07.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.203+0000 7fcafdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3e3857a4-8eff-44cf-b75c-dacf17b4248b", "id": 2}] 2024-03-25T18:25:07.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.203+0000 7fcafdffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:07.205 INFO:teuthology.orchestra.run.smithi016.stdout:2 2024-03-25T18:25:07.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.204+0000 7fcb0f406640 10 librados: watch_flush enter 2024-03-25T18:25:07.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.204+0000 7fcb0f406640 10 librados: watch_flush exit 2024-03-25T18:25:07.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.204+0000 7fcb0f406640 20 client.4131.objecter shutdown clearing up homeless session... 2024-03-25T18:25:07.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.204+0000 7fcb0f406640 10 client.4131.objecter successfully canceled tick 2024-03-25T18:25:07.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.204+0000 7fcb0f406640 10 monclient: shutdown 2024-03-25T18:25:07.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.204+0000 7fcb0f406640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:07.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.204+0000 7fcb0f406640 1 -- 172.21.15.16:0/568119960 >> v1:172.21.15.3:6789/0 conn(0x7fcb0813ff20 legacy=0x7fcb08133e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:07.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.204+0000 7fcb0f406640 1 -- 172.21.15.16:0/568119960 shutdown_connections 2024-03-25T18:25:07.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.204+0000 7fcb0f406640 1 -- 172.21.15.16:0/568119960 >> 172.21.15.16:0/568119960 conn(0x7fcb08066920 msgr2=0x7fcb08066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:07.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.204+0000 7fcb0f406640 1 -- 172.21.15.16:0/568119960 shutdown_connections 2024-03-25T18:25:07.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.204+0000 7fcb0f406640 1 -- 172.21.15.16:0/568119960 wait complete. 2024-03-25T18:25:07.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.204+0000 7fcb0f406640 1 librados: shutdown 2024-03-25T18:25:07.216 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd new 83f74bdb-fb01-41e8-b3da-f69a7459f916 3 2024-03-25T18:25:07.257 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-25T18:25:07.256+0000 7f0c86a58040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:25:07.286 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-03-25T18:25:07.285+0000 7f49c1ff1040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:25:07.304 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-25T18:25:07.303+0000 7f0c86a58040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:25:07.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.315+0000 7ff38f4d9640 10 monclient: get_monmap_and_config 2024-03-25T18:25:07.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.315+0000 7ff38f4d9640 10 monclient: build_initial_monmap 2024-03-25T18:25:07.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.315+0000 7ff38f4d9640 10 monclient: monmap: 2024-03-25T18:25:07.316 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 0 2024-03-25T18:25:07.317 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:07.317 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:25:07.316614+0000 2024-03-25T18:25:07.317 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:25:07.316614+0000 2024-03-25T18:25:07.317 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:07.317 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:07.317 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:07.317 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:25:07.317 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:25:07.317 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:07.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.316+0000 7ff38f4d9640 1 Processor -- start 2024-03-25T18:25:07.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.316+0000 7ff38f4d9640 1 -- start start 2024-03-25T18:25:07.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.316+0000 7ff38f4d9640 10 monclient: init 2024-03-25T18:25:07.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.316+0000 7ff38f4d9640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:07.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.316+0000 7ff38f4d9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:25:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.316+0000 7ff38f4d9640 10 monclient(hunting): picked mon.noname-c con 0x7ff3880b4b50 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.316+0000 7ff38f4d9640 10 monclient(hunting): picked mon.noname-a con 0x7ff38813ff20 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.316+0000 7ff38f4d9640 10 monclient(hunting): picked mon.noname-b con 0x7ff388142dd0 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.316+0000 7ff38f4d9640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3880ab440 con 0x7ff38813ff20 2024-03-25T18:25:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.316+0000 7ff38f4d9640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3880ab6b0 con 0x7ff3880b4b50 2024-03-25T18:25:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.316+0000 7ff38f4d9640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3880a9f10 con 0x7ff388142dd0 2024-03-25T18:25:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.316+0000 7ff38f4d9640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.316+0000 7ff38f4d9640 10 monclient(hunting): authenticate will time out at 7364.452458s 2024-03-25T18:25:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.317+0000 7ff38ca74640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ff38813ff20 0x7ff3881426c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.16:45516/0 (socket says 172.21.15.16:45516) 2024-03-25T18:25:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.317+0000 7ff38ca74640 1 -- 172.21.15.16:0/1497809399 learned_addr learned my addr 172.21.15.16:0/1497809399 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-25T18:25:07.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.320+0000 7ff37ffff640 1 -- 172.21.15.16:0/1497809399 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 189048663 0 0) 0x7ff3880ab440 con 0x7ff38813ff20 2024-03-25T18:25:07.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.320+0000 7ff37ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:07.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.320+0000 7ff37ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:07.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.320+0000 7ff37ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:07.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.320+0000 7ff37ffff640 10 monclient(hunting): my global_id is 4134 2024-03-25T18:25:07.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.320+0000 7ff37ffff640 1 -- 172.21.15.16:0/1497809399 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff374003520 con 0x7ff38813ff20 2024-03-25T18:25:07.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.320+0000 7ff37ffff640 1 -- 172.21.15.16:0/1497809399 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3953971007 0 0) 0x7ff3880ab6b0 con 0x7ff3880b4b50 2024-03-25T18:25:07.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.320+0000 7ff37ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:07.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.320+0000 7ff37ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:07.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.320+0000 7ff37ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:07.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.320+0000 7ff37ffff640 10 monclient(hunting): my global_id is 4126 2024-03-25T18:25:07.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.320+0000 7ff37ffff640 1 -- 172.21.15.16:0/1497809399 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3880ab440 con 0x7ff3880b4b50 2024-03-25T18:25:07.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.320+0000 7ff37ffff640 1 -- 172.21.15.16:0/1497809399 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615173528 0 0) 0x7ff3880a9f10 con 0x7ff388142dd0 2024-03-25T18:25:07.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.320+0000 7ff37ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:07.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.320+0000 7ff37ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:07.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.320+0000 7ff37ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:07.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.320+0000 7ff37ffff640 10 monclient(hunting): my global_id is 4130 2024-03-25T18:25:07.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.320+0000 7ff37ffff640 1 -- 172.21.15.16:0/1497809399 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3880ab6b0 con 0x7ff388142dd0 2024-03-25T18:25:07.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.323+0000 7ff37ffff640 1 -- 172.21.15.16:0/1497809399 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3079727024 0 0) 0x7ff3880ab440 con 0x7ff3880b4b50 2024-03-25T18:25:07.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.323+0000 7ff37ffff640 1 -- 172.21.15.16:0/1497809399 >> v1:172.21.15.3:6790/0 conn(0x7ff388142dd0 legacy=0x7ff38814df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:07.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.323+0000 7ff37ffff640 1 -- 172.21.15.16:0/1497809399 >> v1:172.21.15.3:6789/0 conn(0x7ff38813ff20 legacy=0x7ff3881426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:07.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.323+0000 7ff37ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:25:07.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.323+0000 7ff37ffff640 1 monclient: found mon.noname-c 2024-03-25T18:25:07.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.323+0000 7ff37ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:07.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.323+0000 7ff37ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:25:07.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.323+0000 7ff37ffff640 1 -- 172.21.15.16:0/1497809399 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3880aa0a0 con 0x7ff3880b4b50 2024-03-25T18:25:07.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.323+0000 7ff37ffff640 10 monclient: _finish_auth 0 2024-03-25T18:25:07.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.323+0000 7ff37ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:25:07.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.323+0000 7ff37ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:07.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.323+0000 7ff37ffff640 1 -- 172.21.15.16:0/1497809399 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7ff3840019f0 con 0x7ff3880b4b50 2024-03-25T18:25:07.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.323+0000 7ff38f4d9640 5 monclient: authenticate success, global_id 4126 2024-03-25T18:25:07.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.323+0000 7ff38f4d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:07.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.323+0000 7ff37ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:07.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.323+0000 7ff37ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:07.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.323+0000 7ff37ffff640 10 monclient: dump: 2024-03-25T18:25:07.326 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:07.326 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:07.326 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:07.326 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:07.326 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:07.326 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:07.326 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:07.326 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:07.326 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:07.326 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:07.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.323+0000 7ff38f4d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:07.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.326+0000 7ff37ffff640 1 -- 172.21.15.16:0/1497809399 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff384001b50 con 0x7ff3880b4b50 2024-03-25T18:25:07.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.326+0000 7ff37ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:07.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.326+0000 7ff37ffff640 1 -- 172.21.15.16:0/1497809399 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7ff384001ee0 con 0x7ff3880b4b50 2024-03-25T18:25:07.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.326+0000 7ff38f4d9640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:07.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.326+0000 7ff38f4d9640 4 set_mon_vals no callback set 2024-03-25T18:25:07.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.326+0000 7ff37ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:07.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.326+0000 7ff37ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:07.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.326+0000 7ff37ffff640 10 monclient: dump: 2024-03-25T18:25:07.327 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:07.328 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:07.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.327+0000 7ff38f4d9640 10 monclient: shutdown 2024-03-25T18:25:07.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.327+0000 7ff38f4d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:07.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.327+0000 7ff38f4d9640 1 -- 172.21.15.16:0/1497809399 >> v1:172.21.15.16:6789/0 conn(0x7ff3880b4b50 legacy=0x7ff38813f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:07.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.327+0000 7ff38f4d9640 1 -- 172.21.15.16:0/1497809399 shutdown_connections 2024-03-25T18:25:07.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.327+0000 7ff38f4d9640 1 -- 172.21.15.16:0/1497809399 >> 172.21.15.16:0/1497809399 conn(0x7ff388066920 msgr2=0x7ff388066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:07.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.327+0000 7ff38f4d9640 1 -- 172.21.15.16:0/1497809399 shutdown_connections 2024-03-25T18:25:07.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.327+0000 7ff38f4d9640 1 -- 172.21.15.16:0/1497809399 wait complete. 2024-03-25T18:25:07.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.327+0000 7ff38f4d9640 10 monclient: build_initial_monmap 2024-03-25T18:25:07.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.327+0000 7ff38f4d9640 10 monclient: monmap: 2024-03-25T18:25:07.331 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 0 2024-03-25T18:25:07.331 INFO:teuthology.orchestra.run.smithi016.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:07.331 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 0.000000 2024-03-25T18:25:07.331 INFO:teuthology.orchestra.run.smithi016.stderr:created 0.000000 2024-03-25T18:25:07.331 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:07.331 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 1 2024-03-25T18:25:07.331 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:07.331 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:25:07.332 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:25:07.332 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:07.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.328+0000 7ff38f4d9640 1 librados: starting msgr at 2024-03-25T18:25:07.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.328+0000 7ff38f4d9640 1 librados: starting objecter 2024-03-25T18:25:07.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.328+0000 7ff38f4d9640 1 Processor -- start 2024-03-25T18:25:07.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.328+0000 7ff38f4d9640 1 -- start start 2024-03-25T18:25:07.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.328+0000 7ff38f4d9640 1 librados: setting wanted keys 2024-03-25T18:25:07.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.328+0000 7ff38f4d9640 1 librados: calling monclient init 2024-03-25T18:25:07.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.328+0000 7ff38f4d9640 10 monclient: init 2024-03-25T18:25:07.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.328+0000 7ff38f4d9640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:07.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.328+0000 7ff38f4d9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:07.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.328+0000 7ff38f4d9640 10 monclient(hunting): picked mon.noname-b con 0x7ff3880b4b50 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:07.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.328+0000 7ff38f4d9640 10 monclient(hunting): picked mon.noname-c con 0x7ff38813ff20 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:07.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.328+0000 7ff38f4d9640 10 monclient(hunting): picked mon.noname-a con 0x7ff388142dd0 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:07.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.328+0000 7ff38f4d9640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff388144740 con 0x7ff388142dd0 2024-03-25T18:25:07.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.329+0000 7ff38f4d9640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff38813a670 con 0x7ff3880b4b50 2024-03-25T18:25:07.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.329+0000 7ff38f4d9640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff38813ab90 con 0x7ff38813ff20 2024-03-25T18:25:07.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.329+0000 7ff38f4d9640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:07.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.329+0000 7ff38f4d9640 10 monclient(hunting): authenticate will time out at 7364.464607s 2024-03-25T18:25:07.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.329+0000 7ff38da76640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ff388142dd0 0x7ff3881e3070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.16:45528/0 (socket says 172.21.15.16:45528) 2024-03-25T18:25:07.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.329+0000 7ff38da76640 1 -- 172.21.15.16:0/2114223736 learned_addr learned my addr 172.21.15.16:0/2114223736 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-25T18:25:07.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.329+0000 7ff37dffb640 10 client.?.objecter ms_handle_connect 0x7ff388142dd0 2024-03-25T18:25:07.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.329+0000 7ff37dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:07.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.329+0000 7ff37dffb640 10 client.?.objecter ms_handle_connect 0x7ff3880b4b50 2024-03-25T18:25:07.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.329+0000 7ff37dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:07.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.329+0000 7ff37dffb640 10 client.?.objecter ms_handle_connect 0x7ff38813ff20 2024-03-25T18:25:07.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.329+0000 7ff37dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:07.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.332+0000 7ff37dffb640 1 -- 172.21.15.16:0/2114223736 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2900142566 0 0) 0x7ff388144740 con 0x7ff388142dd0 2024-03-25T18:25:07.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.332+0000 7ff37dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:07.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.332+0000 7ff37dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:07.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.332+0000 7ff37dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:07.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.332+0000 7ff37dffb640 10 monclient(hunting): my global_id is 4137 2024-03-25T18:25:07.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.332+0000 7ff37dffb640 1 -- 172.21.15.16:0/2114223736 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff36c004eb0 con 0x7ff388142dd0 2024-03-25T18:25:07.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.332+0000 7ff37dffb640 1 -- 172.21.15.16:0/2114223736 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600558815 0 0) 0x7ff38813a670 con 0x7ff3880b4b50 2024-03-25T18:25:07.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.332+0000 7ff37dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:07.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.332+0000 7ff37dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:07.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.332+0000 7ff37dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:07.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.332+0000 7ff37dffb640 10 monclient(hunting): my global_id is 4129 2024-03-25T18:25:07.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.332+0000 7ff37dffb640 1 -- 172.21.15.16:0/2114223736 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff388144740 con 0x7ff3880b4b50 2024-03-25T18:25:07.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.332+0000 7ff37dffb640 1 -- 172.21.15.16:0/2114223736 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4059521382 0 0) 0x7ff38813ab90 con 0x7ff38813ff20 2024-03-25T18:25:07.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.332+0000 7ff37dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:07.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.332+0000 7ff37dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:07.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.332+0000 7ff37dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:07.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.332+0000 7ff37dffb640 10 monclient(hunting): my global_id is 4133 2024-03-25T18:25:07.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.332+0000 7ff37dffb640 1 -- 172.21.15.16:0/2114223736 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff38813a670 con 0x7ff38813ff20 2024-03-25T18:25:07.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.337+0000 7ff37dffb640 1 -- 172.21.15.16:0/2114223736 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2184154440 0 0) 0x7ff388144740 con 0x7ff3880b4b50 2024-03-25T18:25:07.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.337+0000 7ff37dffb640 1 -- 172.21.15.16:0/2114223736 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff38813ab90 con 0x7ff3880b4b50 2024-03-25T18:25:07.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.337+0000 7ff37dffb640 1 -- 172.21.15.16:0/2114223736 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7ff384001e60 con 0x7ff3880b4b50 2024-03-25T18:25:07.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.337+0000 7ff37dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:07.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.337+0000 7ff37dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:07.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.337+0000 7ff37dffb640 10 monclient(hunting): dump: 2024-03-25T18:25:07.339 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:07.339 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:07.339 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:07.339 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:07.339 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:07.339 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:07.339 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:07.339 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:07.339 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:07.340 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:07.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.337+0000 7ff37dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:07.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.338+0000 7ff37dffb640 1 -- 172.21.15.16:0/2114223736 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1252891441 0 0) 0x7ff36c004eb0 con 0x7ff388142dd0 2024-03-25T18:25:07.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.338+0000 7ff37dffb640 1 -- 172.21.15.16:0/2114223736 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff388144740 con 0x7ff388142dd0 2024-03-25T18:25:07.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.338+0000 7ff37dffb640 1 -- 172.21.15.16:0/2114223736 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7ff3800018f0 con 0x7ff388142dd0 2024-03-25T18:25:07.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.338+0000 7ff37dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:07.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.338+0000 7ff37dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:07.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.338+0000 7ff37dffb640 10 monclient(hunting): dump: 2024-03-25T18:25:07.340 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:07.340 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:07.340 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:07.340 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:07.340 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:07.340 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:07.341 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:07.341 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:07.341 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:07.341 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:07.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.339+0000 7ff37dffb640 1 -- 172.21.15.16:0/2114223736 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2194284041 0 0) 0x7ff38813a670 con 0x7ff38813ff20 2024-03-25T18:25:07.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.339+0000 7ff37dffb640 1 -- 172.21.15.16:0/2114223736 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff36c004eb0 con 0x7ff38813ff20 2024-03-25T18:25:07.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.339+0000 7ff37dffb640 1 -- 172.21.15.16:0/2114223736 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7ff3780017b0 con 0x7ff38813ff20 2024-03-25T18:25:07.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.339+0000 7ff37dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:07.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.339+0000 7ff37dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:07.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.339+0000 7ff37dffb640 10 monclient(hunting): dump: 2024-03-25T18:25:07.342 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:07.342 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:07.342 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:07.342 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:07.342 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:07.342 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:07.342 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:07.342 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:07.342 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:07.342 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:07.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.341+0000 7ff37dffb640 1 -- 172.21.15.16:0/2114223736 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1133469947 0 0) 0x7ff38813ab90 con 0x7ff3880b4b50 2024-03-25T18:25:07.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.341+0000 7ff37dffb640 1 -- 172.21.15.16:0/2114223736 >> v1:172.21.15.3:6790/0 conn(0x7ff38813ff20 legacy=0x7ff388145f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:07.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.341+0000 7ff37dffb640 1 -- 172.21.15.16:0/2114223736 >> v1:172.21.15.3:6789/0 conn(0x7ff388142dd0 legacy=0x7ff3881e3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:07.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.341+0000 7ff37dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:25:07.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.341+0000 7ff37dffb640 1 monclient: found mon.b 2024-03-25T18:25:07.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.341+0000 7ff37dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:07.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.341+0000 7ff37dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:07.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.341+0000 7ff37dffb640 1 -- 172.21.15.16:0/2114223736 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38813ad20 con 0x7ff3880b4b50 2024-03-25T18:25:07.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.341+0000 7ff37dffb640 10 monclient: _finish_auth 0 2024-03-25T18:25:07.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.341+0000 7ff37dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:25:07.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.341+0000 7ff37dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:07.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.341+0000 7ff38f4d9640 5 monclient: authenticate success, global_id 4129 2024-03-25T18:25:07.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.341+0000 7ff38f4d9640 10 monclient: _renew_subs 2024-03-25T18:25:07.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.341+0000 7ff38f4d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:07.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.341+0000 7ff38f4d9640 1 -- 172.21.15.16:0/2114223736 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff38813a1e0 con 0x7ff3880b4b50 2024-03-25T18:25:07.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.341+0000 7ff38f4d9640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:07.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.341+0000 7ff38f4d9640 10 monclient: _renew_subs 2024-03-25T18:25:07.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.341+0000 7ff38f4d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:07.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.341+0000 7ff38f4d9640 1 -- 172.21.15.16:0/2114223736 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3881d8040 con 0x7ff3880b4b50 2024-03-25T18:25:07.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.341+0000 7ff38f4d9640 1 librados: init done 2024-03-25T18:25:07.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.342+0000 7ff38f4d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:07.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.342+0000 7ff38f4d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:07.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.342+0000 7ff38f4d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:07.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.342+0000 7ff38f4d9640 1 -- 172.21.15.16:0/2114223736 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff38813a8b0 con 0x7ff3880b4b50 2024-03-25T18:25:07.425 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-25T18:25:07.424+0000 7f0c86a58040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:25:07.428 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-03-25T18:25:07.427+0000 7f49c1ff1040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:25:07.475 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-03-25T18:25:07.475+0000 7f49c1ff1040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:25:07.528 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-25T18:25:07.527+0000 7f0c86a58040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:25:07.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.552+0000 7ff37dffb640 1 -- 172.21.15.16:0/2114223736 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff384003c20 con 0x7ff3880b4b50 2024-03-25T18:25:07.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.552+0000 7ff37dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:07.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.552+0000 7ff37dffb640 1 -- 172.21.15.16:0/2114223736 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7ff384003f30 con 0x7ff3880b4b50 2024-03-25T18:25:07.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.552+0000 7ff37dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:07.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.552+0000 7ff37dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:07.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.552+0000 7ff37dffb640 10 monclient: dump: 2024-03-25T18:25:07.553 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:07.554 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:07.554 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:07.554 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:07.554 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:07.554 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:07.554 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:07.554 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:07.554 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:07.554 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:07.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.552+0000 7ff37dffb640 1 -- 172.21.15.16:0/2114223736 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff384004420 con 0x7ff3880b4b50 2024-03-25T18:25:07.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.552+0000 7ff37f7fe640 4 set_mon_vals no callback set 2024-03-25T18:25:07.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.552+0000 7ff37dffb640 1 -- 172.21.15.16:0/2114223736 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 2043427886 0 0) 0x7ff384005940 con 0x7ff3880b4b50 2024-03-25T18:25:07.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.552+0000 7ff37dffb640 10 client.4129.objecter ms_dispatch 0x7ff388000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T18:25:07.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.552+0000 7ff37dffb640 3 client.4129.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T18:25:07.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.552+0000 7ff37dffb640 3 client.4129.objecter handle_osd_map decoding full epoch 5 2024-03-25T18:25:07.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.552+0000 7ff37dffb640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-25T18:25:07.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.552+0000 7ff37dffb640 1 -- 172.21.15.16:0/2114223736 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff384017e00 con 0x7ff3880b4b50 2024-03-25T18:25:07.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.552+0000 7ff37dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:07.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.552+0000 7ff37dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:07.600 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-03-25T18:25:07.599+0000 7f49c1ff1040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:25:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.627+0000 7ff38f4d9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "83f74bdb-fb01-41e8-b3da-f69a7459f916", "id": 3}] 2024-03-25T18:25:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.627+0000 7ff38f4d9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "83f74bdb-fb01-41e8-b3da-f69a7459f916", "id": 3}] 2024-03-25T18:25:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.627+0000 7ff38f4d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.627+0000 7ff38f4d9640 1 -- 172.21.15.16:0/2114223736 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd new", "uuid": "83f74bdb-fb01-41e8-b3da-f69a7459f916", "id": 3} v 0) v1 -- 0x7ff3880b6180 con 0x7ff3880b4b50 2024-03-25T18:25:07.695 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-25T18:25:07.694+0000 7f0c86a58040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:25:07.702 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-03-25T18:25:07.701+0000 7f49c1ff1040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:25:07.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.741+0000 7ff37dffb640 1 -- 172.21.15.16:0/2114223736 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "83f74bdb-fb01-41e8-b3da-f69a7459f916", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 1405519353 0 525046306) 0x7ff3840053f0 con 0x7ff3880b4b50 2024-03-25T18:25:07.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.741+0000 7ff37dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "83f74bdb-fb01-41e8-b3da-f69a7459f916", "id": 3}] 2024-03-25T18:25:07.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.741+0000 7ff37dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:07.743 INFO:teuthology.orchestra.run.smithi016.stdout:3 2024-03-25T18:25:07.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.742+0000 7ff38f4d9640 10 librados: watch_flush enter 2024-03-25T18:25:07.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.742+0000 7ff38f4d9640 10 librados: watch_flush exit 2024-03-25T18:25:07.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.742+0000 7ff38f4d9640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-25T18:25:07.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.742+0000 7ff38f4d9640 10 client.4129.objecter successfully canceled tick 2024-03-25T18:25:07.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.742+0000 7ff38f4d9640 10 monclient: shutdown 2024-03-25T18:25:07.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.742+0000 7ff38f4d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:07.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.742+0000 7ff38f4d9640 1 -- 172.21.15.16:0/2114223736 >> v1:172.21.15.16:6789/0 conn(0x7ff3880b4b50 legacy=0x7ff3881d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:07.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.742+0000 7ff38f4d9640 1 -- 172.21.15.16:0/2114223736 shutdown_connections 2024-03-25T18:25:07.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.743+0000 7ff38f4d9640 1 -- 172.21.15.16:0/2114223736 >> 172.21.15.16:0/2114223736 conn(0x7ff388066920 msgr2=0x7ff388066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:07.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.743+0000 7ff38f4d9640 1 -- 172.21.15.16:0/2114223736 shutdown_connections 2024-03-25T18:25:07.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.743+0000 7ff38f4d9640 1 -- 172.21.15.16:0/2114223736 wait complete. 2024-03-25T18:25:07.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.743+0000 7ff38f4d9640 1 librados: shutdown 2024-03-25T18:25:07.746 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-25T18:25:07.742+0000 7f0c86a58040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:25:07.754 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd new a765e52f-f1cf-4ad6-8f18-6df9db782869 4 2024-03-25T18:25:07.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.853+0000 7fc48821e640 10 monclient: get_monmap_and_config 2024-03-25T18:25:07.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.853+0000 7fc48821e640 10 monclient: build_initial_monmap 2024-03-25T18:25:07.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.853+0000 7fc48821e640 10 monclient: monmap: 2024-03-25T18:25:07.854 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 0 2024-03-25T18:25:07.854 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:07.854 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:25:07.854130+0000 2024-03-25T18:25:07.854 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:25:07.854130+0000 2024-03-25T18:25:07.854 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:07.854 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:07.855 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:07.855 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:25:07.855 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:25:07.855 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:07.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.853+0000 7fc48821e640 1 Processor -- start 2024-03-25T18:25:07.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.853+0000 7fc48821e640 1 -- start start 2024-03-25T18:25:07.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.853+0000 7fc48821e640 10 monclient: init 2024-03-25T18:25:07.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.854+0000 7fc48821e640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:07.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.854+0000 7fc48821e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:07.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.854+0000 7fc48821e640 10 monclient(hunting): picked mon.noname-a con 0x7fc4800b4b50 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:07.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.854+0000 7fc48821e640 10 monclient(hunting): picked mon.noname-b con 0x7fc48013ff20 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:07.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.854+0000 7fc48821e640 10 monclient(hunting): picked mon.noname-c con 0x7fc480142dd0 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:07.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.854+0000 7fc48821e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4800ab440 con 0x7fc4800b4b50 2024-03-25T18:25:07.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.854+0000 7fc48821e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4800ab6b0 con 0x7fc480142dd0 2024-03-25T18:25:07.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.854+0000 7fc48821e640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4800a9f10 con 0x7fc48013ff20 2024-03-25T18:25:07.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.854+0000 7fc48821e640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:07.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.854+0000 7fc48821e640 10 monclient(hunting): authenticate will time out at 7364.990107s 2024-03-25T18:25:07.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.855+0000 7fc485fba640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fc4800b4b50 0x7fc48013f810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.16:45534/0 (socket says 172.21.15.16:45534) 2024-03-25T18:25:07.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.855+0000 7fc485fba640 1 -- 172.21.15.16:0/2546105681 learned_addr learned my addr 172.21.15.16:0/2546105681 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-25T18:25:07.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.857+0000 7fc484fb8640 1 -- 172.21.15.16:0/2546105681 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2376359357 0 0) 0x7fc4800ab440 con 0x7fc4800b4b50 2024-03-25T18:25:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.857+0000 7fc484fb8640 10 monclient(hunting): _negotiate 2024-03-25T18:25:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.857+0000 7fc484fb8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.857+0000 7fc484fb8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.857+0000 7fc484fb8640 10 monclient(hunting): my global_id is 4140 2024-03-25T18:25:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.857+0000 7fc484fb8640 1 -- 172.21.15.16:0/2546105681 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc46c003520 con 0x7fc4800b4b50 2024-03-25T18:25:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.857+0000 7fc484fb8640 1 -- 172.21.15.16:0/2546105681 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1800746248 0 0) 0x7fc4800ab6b0 con 0x7fc480142dd0 2024-03-25T18:25:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.857+0000 7fc484fb8640 10 monclient(hunting): _negotiate 2024-03-25T18:25:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.857+0000 7fc484fb8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.857+0000 7fc484fb8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.857+0000 7fc484fb8640 10 monclient(hunting): my global_id is 4132 2024-03-25T18:25:07.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.857+0000 7fc484fb8640 1 -- 172.21.15.16:0/2546105681 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4800ab440 con 0x7fc480142dd0 2024-03-25T18:25:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.859+0000 7fc484fb8640 1 -- 172.21.15.16:0/2546105681 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2300523434 0 0) 0x7fc4800a9f10 con 0x7fc48013ff20 2024-03-25T18:25:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.859+0000 7fc484fb8640 10 monclient(hunting): _negotiate 2024-03-25T18:25:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.859+0000 7fc484fb8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.859+0000 7fc484fb8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.859+0000 7fc484fb8640 10 monclient(hunting): my global_id is 4136 2024-03-25T18:25:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.859+0000 7fc484fb8640 1 -- 172.21.15.16:0/2546105681 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4800ab6b0 con 0x7fc48013ff20 2024-03-25T18:25:07.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.861+0000 7fc484fb8640 1 -- 172.21.15.16:0/2546105681 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 962005119 0 0) 0x7fc4800ab440 con 0x7fc480142dd0 2024-03-25T18:25:07.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.861+0000 7fc484fb8640 1 -- 172.21.15.16:0/2546105681 >> v1:172.21.15.3:6790/0 conn(0x7fc48013ff20 legacy=0x7fc4801426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.861+0000 7fc484fb8640 1 -- 172.21.15.16:0/2546105681 >> v1:172.21.15.3:6789/0 conn(0x7fc4800b4b50 legacy=0x7fc48013f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.861+0000 7fc484fb8640 10 monclient: _finish_hunting 0 2024-03-25T18:25:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.861+0000 7fc484fb8640 1 monclient: found mon.noname-c 2024-03-25T18:25:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.861+0000 7fc484fb8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.861+0000 7fc484fb8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:25:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.861+0000 7fc484fb8640 1 -- 172.21.15.16:0/2546105681 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4800aa0a0 con 0x7fc480142dd0 2024-03-25T18:25:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.861+0000 7fc484fb8640 10 monclient: _finish_auth 0 2024-03-25T18:25:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.861+0000 7fc484fb8640 10 monclient: _check_auth_tickets 2024-03-25T18:25:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.861+0000 7fc484fb8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.861+0000 7fc48821e640 5 monclient: authenticate success, global_id 4132 2024-03-25T18:25:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.861+0000 7fc48821e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.861+0000 7fc484fb8640 1 -- 172.21.15.16:0/2546105681 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc478001930 con 0x7fc480142dd0 2024-03-25T18:25:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.861+0000 7fc484fb8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.861+0000 7fc484fb8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:07.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.861+0000 7fc484fb8640 10 monclient: dump: 2024-03-25T18:25:07.864 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:07.864 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:07.864 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:07.864 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:07.864 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:07.864 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:07.864 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:07.864 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:07.864 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:07.864 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:07.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.861+0000 7fc48821e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:07.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.863+0000 7fc484fb8640 1 -- 172.21.15.16:0/2546105681 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc478001b10 con 0x7fc480142dd0 2024-03-25T18:25:07.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.863+0000 7fc484fb8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:07.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.863+0000 7fc484fb8640 1 -- 172.21.15.16:0/2546105681 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc478001ea0 con 0x7fc480142dd0 2024-03-25T18:25:07.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.863+0000 7fc48821e640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:07.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.863+0000 7fc48821e640 4 set_mon_vals no callback set 2024-03-25T18:25:07.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.863+0000 7fc484fb8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:07.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.863+0000 7fc484fb8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:07.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.863+0000 7fc484fb8640 10 monclient: dump: 2024-03-25T18:25:07.868 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:07.868 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:07.868 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:07.868 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:07.868 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:07.868 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:07.868 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:07.868 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:07.868 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:07.868 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:07.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.863+0000 7fc48821e640 10 monclient: shutdown 2024-03-25T18:25:07.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.863+0000 7fc48821e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:07.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.863+0000 7fc48821e640 1 -- 172.21.15.16:0/2546105681 >> v1:172.21.15.16:6789/0 conn(0x7fc480142dd0 legacy=0x7fc48014df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:07.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.863+0000 7fc48821e640 1 -- 172.21.15.16:0/2546105681 shutdown_connections 2024-03-25T18:25:07.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.863+0000 7fc48821e640 1 -- 172.21.15.16:0/2546105681 >> 172.21.15.16:0/2546105681 conn(0x7fc480066920 msgr2=0x7fc480066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:07.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.863+0000 7fc48821e640 1 -- 172.21.15.16:0/2546105681 shutdown_connections 2024-03-25T18:25:07.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.863+0000 7fc48821e640 1 -- 172.21.15.16:0/2546105681 wait complete. 2024-03-25T18:25:07.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.864+0000 7fc48821e640 10 monclient: build_initial_monmap 2024-03-25T18:25:07.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.864+0000 7fc48821e640 10 monclient: monmap: 2024-03-25T18:25:07.869 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 0 2024-03-25T18:25:07.869 INFO:teuthology.orchestra.run.smithi016.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:07.869 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 0.000000 2024-03-25T18:25:07.869 INFO:teuthology.orchestra.run.smithi016.stderr:created 0.000000 2024-03-25T18:25:07.869 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:07.869 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 1 2024-03-25T18:25:07.869 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:07.869 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:25:07.869 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:25:07.869 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:07.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.865+0000 7fc48821e640 1 librados: starting msgr at 2024-03-25T18:25:07.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.865+0000 7fc48821e640 1 librados: starting objecter 2024-03-25T18:25:07.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.865+0000 7fc48821e640 1 Processor -- start 2024-03-25T18:25:07.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.865+0000 7fc48821e640 1 -- start start 2024-03-25T18:25:07.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.865+0000 7fc48821e640 1 librados: setting wanted keys 2024-03-25T18:25:07.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.865+0000 7fc48821e640 1 librados: calling monclient init 2024-03-25T18:25:07.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.865+0000 7fc48821e640 10 monclient: init 2024-03-25T18:25:07.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.865+0000 7fc48821e640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:07.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.865+0000 7fc48821e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:25:07.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.865+0000 7fc48821e640 10 monclient(hunting): picked mon.noname-b con 0x7fc4800b4b50 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:07.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.865+0000 7fc48821e640 10 monclient(hunting): picked mon.noname-a con 0x7fc48013ff20 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:07.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.865+0000 7fc48821e640 10 monclient(hunting): picked mon.noname-c con 0x7fc480142dd0 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:07.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.865+0000 7fc48821e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4801439b0 con 0x7fc48013ff20 2024-03-25T18:25:07.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.865+0000 7fc48821e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc480143b40 con 0x7fc4800b4b50 2024-03-25T18:25:07.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.865+0000 7fc48821e640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc48013eae0 con 0x7fc480142dd0 2024-03-25T18:25:07.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.865+0000 7fc48821e640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:07.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.865+0000 7fc48821e640 10 monclient(hunting): authenticate will time out at 7365.001317s 2024-03-25T18:25:07.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.866+0000 7fc485fba640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc4800b4b50 0x7fc48013a9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37794/0 (socket says 172.21.15.16:37794) 2024-03-25T18:25:07.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.866+0000 7fc485fba640 1 -- 172.21.15.16:0/2363369447 learned_addr learned my addr 172.21.15.16:0/2363369447 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-25T18:25:07.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.866+0000 7fc4857b9640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fc48013ff20 0x7fc480139b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.16:45546/0 (socket says 172.21.15.16:45546) 2024-03-25T18:25:07.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.866+0000 7fc4727fc640 10 client.?.objecter ms_handle_connect 0x7fc4800b4b50 2024-03-25T18:25:07.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.866+0000 7fc4727fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:07.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.866+0000 7fc4727fc640 10 client.?.objecter ms_handle_connect 0x7fc48013ff20 2024-03-25T18:25:07.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.866+0000 7fc4727fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:07.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.866+0000 7fc4727fc640 10 client.?.objecter ms_handle_connect 0x7fc480142dd0 2024-03-25T18:25:07.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.866+0000 7fc4727fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:07.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.868+0000 7fc4727fc640 1 -- 172.21.15.16:0/2363369447 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3415191426 0 0) 0x7fc480143b40 con 0x7fc4800b4b50 2024-03-25T18:25:07.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.868+0000 7fc4727fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:07.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.868+0000 7fc4727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:07.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.868+0000 7fc4727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:07.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.868+0000 7fc4727fc640 10 monclient(hunting): my global_id is 4135 2024-03-25T18:25:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.868+0000 7fc4727fc640 1 -- 172.21.15.16:0/2363369447 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc464004eb0 con 0x7fc4800b4b50 2024-03-25T18:25:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.868+0000 7fc4727fc640 1 -- 172.21.15.16:0/2363369447 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 89685125 0 0) 0x7fc4801439b0 con 0x7fc48013ff20 2024-03-25T18:25:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.868+0000 7fc4727fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.868+0000 7fc4727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.868+0000 7fc4727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.868+0000 7fc4727fc640 10 monclient(hunting): my global_id is 4143 2024-03-25T18:25:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.868+0000 7fc4727fc640 1 -- 172.21.15.16:0/2363369447 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc480143b40 con 0x7fc48013ff20 2024-03-25T18:25:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.869+0000 7fc4727fc640 1 -- 172.21.15.16:0/2363369447 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588052008 0 0) 0x7fc48013eae0 con 0x7fc480142dd0 2024-03-25T18:25:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.869+0000 7fc4727fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.869+0000 7fc4727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.869+0000 7fc4727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.869+0000 7fc4727fc640 10 monclient(hunting): my global_id is 4139 2024-03-25T18:25:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.869+0000 7fc4727fc640 1 -- 172.21.15.16:0/2363369447 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4801439b0 con 0x7fc480142dd0 2024-03-25T18:25:07.874 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-03-25T18:25:07.869+0000 7f49c1ff1040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:25:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.873+0000 7fc4727fc640 1 -- 172.21.15.16:0/2363369447 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3106604533 0 0) 0x7fc464004eb0 con 0x7fc4800b4b50 2024-03-25T18:25:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.873+0000 7fc4727fc640 1 -- 172.21.15.16:0/2363369447 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc48013eae0 con 0x7fc4800b4b50 2024-03-25T18:25:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.873+0000 7fc4727fc640 1 -- 172.21.15.16:0/2363369447 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc47c001950 con 0x7fc4800b4b50 2024-03-25T18:25:07.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.873+0000 7fc4727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:07.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.873+0000 7fc4727fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:07.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.873+0000 7fc4727fc640 10 monclient(hunting): dump: 2024-03-25T18:25:07.875 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:07.875 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:07.875 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:07.875 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:07.875 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:07.875 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:07.875 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:07.875 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:07.875 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:07.875 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:07.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.873+0000 7fc4727fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:07.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.874+0000 7fc4727fc640 1 -- 172.21.15.16:0/2363369447 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 343983918 0 0) 0x7fc480143b40 con 0x7fc48013ff20 2024-03-25T18:25:07.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.874+0000 7fc4727fc640 1 -- 172.21.15.16:0/2363369447 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc464004eb0 con 0x7fc48013ff20 2024-03-25T18:25:07.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.874+0000 7fc4727fc640 1 -- 172.21.15.16:0/2363369447 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc4740018f0 con 0x7fc48013ff20 2024-03-25T18:25:07.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.874+0000 7fc4727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:07.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.874+0000 7fc4727fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:07.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.874+0000 7fc4727fc640 10 monclient(hunting): dump: 2024-03-25T18:25:07.876 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:07.877 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:07.877 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:07.877 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:07.877 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:07.877 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:07.877 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:07.877 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:07.877 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:07.877 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:07.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.876+0000 7fc4727fc640 1 -- 172.21.15.16:0/2363369447 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2095189726 0 0) 0x7fc4801439b0 con 0x7fc480142dd0 2024-03-25T18:25:07.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.876+0000 7fc4727fc640 1 -- 172.21.15.16:0/2363369447 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc480143b40 con 0x7fc480142dd0 2024-03-25T18:25:07.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.876+0000 7fc4727fc640 1 -- 172.21.15.16:0/2363369447 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc478001e20 con 0x7fc480142dd0 2024-03-25T18:25:07.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.876+0000 7fc4727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:07.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.876+0000 7fc4727fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:07.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.876+0000 7fc4727fc640 10 monclient(hunting): dump: 2024-03-25T18:25:07.879 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:07.879 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:07.879 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:07.879 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:07.879 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:07.879 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:07.879 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:07.880 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.876+0000 7fc4727fc640 1 -- 172.21.15.16:0/2363369447 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2918706223 0 0) 0x7fc48013eae0 con 0x7fc4800b4b50 2024-03-25T18:25:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.876+0000 7fc4727fc640 1 -- 172.21.15.16:0/2363369447 >> v1:172.21.15.3:6790/0 conn(0x7fc480142dd0 legacy=0x7fc4801e2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.876+0000 7fc4727fc640 1 -- 172.21.15.16:0/2363369447 >> v1:172.21.15.3:6789/0 conn(0x7fc48013ff20 legacy=0x7fc480139b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.876+0000 7fc4727fc640 10 monclient: _finish_hunting 0 2024-03-25T18:25:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.876+0000 7fc4727fc640 1 monclient: found mon.b 2024-03-25T18:25:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.876+0000 7fc4727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.876+0000 7fc4727fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.876+0000 7fc4727fc640 1 -- 172.21.15.16:0/2363369447 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48013ec70 con 0x7fc4800b4b50 2024-03-25T18:25:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.876+0000 7fc4727fc640 10 monclient: _finish_auth 0 2024-03-25T18:25:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.876+0000 7fc4727fc640 10 monclient: _check_auth_tickets 2024-03-25T18:25:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.876+0000 7fc4727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.876+0000 7fc48821e640 5 monclient: authenticate success, global_id 4135 2024-03-25T18:25:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.876+0000 7fc48821e640 10 monclient: _renew_subs 2024-03-25T18:25:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.876+0000 7fc48821e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.876+0000 7fc48821e640 1 -- 172.21.15.16:0/2363369447 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc480137f90 con 0x7fc4800b4b50 2024-03-25T18:25:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.876+0000 7fc48821e640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.876+0000 7fc48821e640 10 monclient: _renew_subs 2024-03-25T18:25:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.876+0000 7fc48821e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.876+0000 7fc48821e640 1 -- 172.21.15.16:0/2363369447 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4801e3b60 con 0x7fc4800b4b50 2024-03-25T18:25:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.876+0000 7fc48821e640 1 librados: init done 2024-03-25T18:25:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.877+0000 7fc48821e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.877+0000 7fc48821e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.877+0000 7fc48821e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.877+0000 7fc48821e640 1 -- 172.21.15.16:0/2363369447 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc48013a2a0 con 0x7fc4800b4b50 2024-03-25T18:25:07.909 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-25T18:25:07.908+0000 7f0c86a58040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:25:07.915 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-03-25T18:25:07.914+0000 7f49c1ff1040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:25:07.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.982+0000 7fc4727fc640 1 -- 172.21.15.16:0/2363369447 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc47c0047e0 con 0x7fc4800b4b50 2024-03-25T18:25:07.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.982+0000 7fc4727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:07.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.982+0000 7fc4727fc640 1 -- 172.21.15.16:0/2363369447 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc47c004af0 con 0x7fc4800b4b50 2024-03-25T18:25:07.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.982+0000 7fc4727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:07.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.982+0000 7fc4727fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:07.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.982+0000 7fc4727fc640 10 monclient: dump: 2024-03-25T18:25:07.984 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:07.984 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:07.984 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:07.984 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:07.984 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:07.984 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:07.984 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:07.984 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:07.984 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:07.984 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:07.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.982+0000 7fc473fff640 4 set_mon_vals no callback set 2024-03-25T18:25:07.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.982+0000 7fc4727fc640 1 -- 172.21.15.16:0/2363369447 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc47c004fe0 con 0x7fc4800b4b50 2024-03-25T18:25:07.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.982+0000 7fc4727fc640 1 -- 172.21.15.16:0/2363369447 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 3080562671 0 0) 0x7fc47c006740 con 0x7fc4800b4b50 2024-03-25T18:25:07.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.982+0000 7fc4727fc640 10 client.4135.objecter ms_dispatch 0x7fc480000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T18:25:07.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.982+0000 7fc4727fc640 3 client.4135.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T18:25:07.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.982+0000 7fc4727fc640 3 client.4135.objecter handle_osd_map decoding full epoch 6 2024-03-25T18:25:07.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.982+0000 7fc4727fc640 20 client.4135.objecter dump_active .. 0 homeless 2024-03-25T18:25:07.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.982+0000 7fc4727fc640 1 -- 172.21.15.16:0/2363369447 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc47c006b20 con 0x7fc4800b4b50 2024-03-25T18:25:07.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.982+0000 7fc4727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:07.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:07.982+0000 7fc4727fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:08.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.056+0000 7fc48821e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a765e52f-f1cf-4ad6-8f18-6df9db782869", "id": 4}] 2024-03-25T18:25:08.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.056+0000 7fc48821e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a765e52f-f1cf-4ad6-8f18-6df9db782869", "id": 4}] 2024-03-25T18:25:08.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.056+0000 7fc48821e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:08.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.056+0000 7fc48821e640 1 -- 172.21.15.16:0/2363369447 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd new", "uuid": "a765e52f-f1cf-4ad6-8f18-6df9db782869", "id": 4} v 0) v1 -- 0x7fc48013ee60 con 0x7fc4800b4b50 2024-03-25T18:25:08.076 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-03-25T18:25:08.076+0000 7f49c1ff1040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:25:08.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.142+0000 7fc4727fc640 1 -- 172.21.15.16:0/2363369447 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a765e52f-f1cf-4ad6-8f18-6df9db782869", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 568529292 0 1697012583) 0x7fc47c006220 con 0x7fc4800b4b50 2024-03-25T18:25:08.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.142+0000 7fc4727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a765e52f-f1cf-4ad6-8f18-6df9db782869", "id": 4}] 2024-03-25T18:25:08.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.142+0000 7fc4727fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:08.144 INFO:teuthology.orchestra.run.smithi016.stdout:4 2024-03-25T18:25:08.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.143+0000 7fc48821e640 10 librados: watch_flush enter 2024-03-25T18:25:08.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.143+0000 7fc48821e640 10 librados: watch_flush exit 2024-03-25T18:25:08.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.143+0000 7fc48821e640 20 client.4135.objecter shutdown clearing up homeless session... 2024-03-25T18:25:08.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.143+0000 7fc48821e640 10 client.4135.objecter successfully canceled tick 2024-03-25T18:25:08.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.143+0000 7fc48821e640 10 monclient: shutdown 2024-03-25T18:25:08.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.143+0000 7fc48821e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:08.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.143+0000 7fc48821e640 1 -- 172.21.15.16:0/2363369447 >> v1:172.21.15.16:6789/0 conn(0x7fc4800b4b50 legacy=0x7fc48013a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:08.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.143+0000 7fc48821e640 1 -- 172.21.15.16:0/2363369447 shutdown_connections 2024-03-25T18:25:08.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.143+0000 7fc48821e640 1 -- 172.21.15.16:0/2363369447 >> 172.21.15.16:0/2363369447 conn(0x7fc480066920 msgr2=0x7fc4800b6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:08.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.144+0000 7fc48821e640 1 -- 172.21.15.16:0/2363369447 shutdown_connections 2024-03-25T18:25:08.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.144+0000 7fc48821e640 1 -- 172.21.15.16:0/2363369447 wait complete. 2024-03-25T18:25:08.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.144+0000 7fc48821e640 1 librados: shutdown 2024-03-25T18:25:08.155 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd new 9ca8edca-f60e-4490-bc1d-782bc08c5b09 5 2024-03-25T18:25:08.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.253+0000 7f16309d6640 10 monclient: get_monmap_and_config 2024-03-25T18:25:08.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.253+0000 7f16309d6640 10 monclient: build_initial_monmap 2024-03-25T18:25:08.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.253+0000 7f16309d6640 10 monclient: monmap: 2024-03-25T18:25:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 0 2024-03-25T18:25:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:25:08.254887+0000 2024-03-25T18:25:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:25:08.254887+0000 2024-03-25T18:25:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:25:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:25:08.255 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:08.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.254+0000 7f16309d6640 1 Processor -- start 2024-03-25T18:25:08.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.254+0000 7f16309d6640 1 -- start start 2024-03-25T18:25:08.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.254+0000 7f16309d6640 10 monclient: init 2024-03-25T18:25:08.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.255+0000 7f16309d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:08.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.255+0000 7f16309d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:08.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.255+0000 7f16309d6640 10 monclient(hunting): picked mon.noname-b con 0x7f162c0b4b50 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:08.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.255+0000 7f16309d6640 10 monclient(hunting): picked mon.noname-a con 0x7f162c13ff20 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:08.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.255+0000 7f16309d6640 10 monclient(hunting): picked mon.noname-c con 0x7f162c142dd0 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:08.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.255+0000 7f16309d6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f162c0ab440 con 0x7f162c13ff20 2024-03-25T18:25:08.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.255+0000 7f16309d6640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f162c0ab6b0 con 0x7f162c142dd0 2024-03-25T18:25:08.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.255+0000 7f16309d6640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f162c0a9f10 con 0x7f162c0b4b50 2024-03-25T18:25:08.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.255+0000 7f16309d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:08.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.255+0000 7f16309d6640 10 monclient(hunting): authenticate will time out at 7365.390736s 2024-03-25T18:25:08.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.255+0000 7f1623fff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f162c13ff20 0x7f162c1426c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.16:45556/0 (socket says 172.21.15.16:45556) 2024-03-25T18:25:08.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.255+0000 7f1623fff640 1 -- 172.21.15.16:0/4016145842 learned_addr learned my addr 172.21.15.16:0/4016145842 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-25T18:25:08.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.257+0000 7f1629d9b640 1 -- 172.21.15.16:0/4016145842 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3261572706 0 0) 0x7f162c0ab440 con 0x7f162c13ff20 2024-03-25T18:25:08.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.257+0000 7f1629d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:25:08.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.257+0000 7f1629d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:08.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.257+0000 7f1629d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:08.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.257+0000 7f1629d9b640 10 monclient(hunting): my global_id is 4146 2024-03-25T18:25:08.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.257+0000 7f1629d9b640 1 -- 172.21.15.16:0/4016145842 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1618003520 con 0x7f162c13ff20 2024-03-25T18:25:08.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.258+0000 7f1629d9b640 1 -- 172.21.15.16:0/4016145842 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2481460164 0 0) 0x7f162c0ab6b0 con 0x7f162c142dd0 2024-03-25T18:25:08.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.258+0000 7f1629d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:25:08.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.258+0000 7f1629d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:08.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.258+0000 7f1629d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:08.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.258+0000 7f1629d9b640 10 monclient(hunting): my global_id is 4138 2024-03-25T18:25:08.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.258+0000 7f1629d9b640 1 -- 172.21.15.16:0/4016145842 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f162c0ab440 con 0x7f162c142dd0 2024-03-25T18:25:08.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.259+0000 7f1629d9b640 1 -- 172.21.15.16:0/4016145842 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1305127505 0 0) 0x7f162c0a9f10 con 0x7f162c0b4b50 2024-03-25T18:25:08.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.259+0000 7f1629d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:25:08.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.259+0000 7f1629d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:08.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.259+0000 7f1629d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:08.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.259+0000 7f1629d9b640 10 monclient(hunting): my global_id is 4142 2024-03-25T18:25:08.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.259+0000 7f1629d9b640 1 -- 172.21.15.16:0/4016145842 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f162c0ab6b0 con 0x7f162c0b4b50 2024-03-25T18:25:08.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.261+0000 7f1629d9b640 1 -- 172.21.15.16:0/4016145842 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 679696550 0 0) 0x7f1618003520 con 0x7f162c13ff20 2024-03-25T18:25:08.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.261+0000 7f1629d9b640 1 -- 172.21.15.16:0/4016145842 >> v1:172.21.15.3:6790/0 conn(0x7f162c0b4b50 legacy=0x7f162c13f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:08.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.261+0000 7f1629d9b640 1 -- 172.21.15.16:0/4016145842 >> v1:172.21.15.16:6789/0 conn(0x7f162c142dd0 legacy=0x7f162c14df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:08.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.261+0000 7f1629d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:25:08.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.261+0000 7f1629d9b640 1 monclient: found mon.noname-a 2024-03-25T18:25:08.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.261+0000 7f1629d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:08.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.261+0000 7f1629d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:25:08.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.261+0000 7f1629d9b640 1 -- 172.21.15.16:0/4016145842 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f162c0aa0a0 con 0x7f162c13ff20 2024-03-25T18:25:08.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.261+0000 7f1629d9b640 10 monclient: _finish_auth 0 2024-03-25T18:25:08.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.261+0000 7f1629d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:25:08.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.261+0000 7f1629d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:08.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.261+0000 7f1629d9b640 1 -- 172.21.15.16:0/4016145842 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f1614001690 con 0x7f162c13ff20 2024-03-25T18:25:08.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.261+0000 7f1629d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:08.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.261+0000 7f1629d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:08.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.261+0000 7f1629d9b640 10 monclient: dump: 2024-03-25T18:25:08.263 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:08.263 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:08.263 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:08.263 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:08.263 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:08.263 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:08.264 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:08.264 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:08.264 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:08.264 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:08.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.261+0000 7f16309d6640 5 monclient: authenticate success, global_id 4146 2024-03-25T18:25:08.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.261+0000 7f16309d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:08.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.263+0000 7f1629d9b640 1 -- 172.21.15.16:0/4016145842 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16140017f0 con 0x7f162c13ff20 2024-03-25T18:25:08.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.263+0000 7f1629d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:08.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.263+0000 7f1629d9b640 1 -- 172.21.15.16:0/4016145842 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f1614001b80 con 0x7f162c13ff20 2024-03-25T18:25:08.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.263+0000 7f16309d6640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:08.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.263+0000 7f16309d6640 4 set_mon_vals no callback set 2024-03-25T18:25:08.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.263+0000 7f1629d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:08.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.263+0000 7f1629d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:08.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.263+0000 7f1629d9b640 10 monclient: dump: 2024-03-25T18:25:08.267 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:08.268 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:08.268 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:08.268 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:08.268 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:08.268 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:08.268 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:08.268 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:08.268 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:08.268 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:08.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.263+0000 7f16309d6640 10 monclient: shutdown 2024-03-25T18:25:08.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.263+0000 7f16309d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:08.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.263+0000 7f16309d6640 1 -- 172.21.15.16:0/4016145842 >> v1:172.21.15.3:6789/0 conn(0x7f162c13ff20 legacy=0x7f162c1426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:08.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.263+0000 7f16309d6640 1 -- 172.21.15.16:0/4016145842 shutdown_connections 2024-03-25T18:25:08.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.263+0000 7f16309d6640 1 -- 172.21.15.16:0/4016145842 >> 172.21.15.16:0/4016145842 conn(0x7f162c066920 msgr2=0x7f162c066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:08.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.263+0000 7f16309d6640 1 -- 172.21.15.16:0/4016145842 shutdown_connections 2024-03-25T18:25:08.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.263+0000 7f16309d6640 1 -- 172.21.15.16:0/4016145842 wait complete. 2024-03-25T18:25:08.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.264+0000 7f16309d6640 10 monclient: build_initial_monmap 2024-03-25T18:25:08.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.264+0000 7f16309d6640 10 monclient: monmap: 2024-03-25T18:25:08.269 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 0 2024-03-25T18:25:08.269 INFO:teuthology.orchestra.run.smithi016.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:08.269 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 0.000000 2024-03-25T18:25:08.269 INFO:teuthology.orchestra.run.smithi016.stderr:created 0.000000 2024-03-25T18:25:08.269 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:08.269 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 1 2024-03-25T18:25:08.269 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:08.269 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:25:08.269 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:25:08.269 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:08.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.264+0000 7f16309d6640 1 librados: starting msgr at 2024-03-25T18:25:08.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.264+0000 7f16309d6640 1 librados: starting objecter 2024-03-25T18:25:08.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.265+0000 7f16309d6640 1 Processor -- start 2024-03-25T18:25:08.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.265+0000 7f16309d6640 1 -- start start 2024-03-25T18:25:08.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.265+0000 7f16309d6640 1 librados: setting wanted keys 2024-03-25T18:25:08.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.265+0000 7f16309d6640 1 librados: calling monclient init 2024-03-25T18:25:08.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.265+0000 7f16309d6640 10 monclient: init 2024-03-25T18:25:08.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.265+0000 7f16309d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:08.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.265+0000 7f16309d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:08.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.265+0000 7f16309d6640 10 monclient(hunting): picked mon.noname-a con 0x7f162c0b4b50 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:08.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.265+0000 7f16309d6640 10 monclient(hunting): picked mon.noname-c con 0x7f162c13ff20 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:08.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.265+0000 7f16309d6640 10 monclient(hunting): picked mon.noname-b con 0x7f162c142dd0 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:08.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.265+0000 7f16309d6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f162c145e80 con 0x7f162c0b4b50 2024-03-25T18:25:08.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.265+0000 7f16309d6640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f162c144b50 con 0x7f162c142dd0 2024-03-25T18:25:08.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.265+0000 7f16309d6640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f162c146310 con 0x7f162c13ff20 2024-03-25T18:25:08.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.265+0000 7f16309d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:08.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.265+0000 7f16309d6640 10 monclient(hunting): authenticate will time out at 7365.401283s 2024-03-25T18:25:08.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.266+0000 7f162a59c640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f162c0b4b50 0x7f162c137110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.16:45560/0 (socket says 172.21.15.16:45560) 2024-03-25T18:25:08.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.266+0000 7f162a59c640 1 -- 172.21.15.16:0/3516323560 learned_addr learned my addr 172.21.15.16:0/3516323560 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-25T18:25:08.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.266+0000 7f1622ffd640 10 client.?.objecter ms_handle_connect 0x7f162c0b4b50 2024-03-25T18:25:08.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.266+0000 7f1622ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:08.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.266+0000 7f1622ffd640 10 client.?.objecter ms_handle_connect 0x7f162c142dd0 2024-03-25T18:25:08.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.266+0000 7f1622ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:08.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.266+0000 7f1622ffd640 10 client.?.objecter ms_handle_connect 0x7f162c13ff20 2024-03-25T18:25:08.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.266+0000 7f1622ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:08.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.268+0000 7f1622ffd640 1 -- 172.21.15.16:0/3516323560 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308928794 0 0) 0x7f162c145e80 con 0x7f162c0b4b50 2024-03-25T18:25:08.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.268+0000 7f1622ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:08.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.268+0000 7f1622ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:08.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.268+0000 7f1622ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:08.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.268+0000 7f1622ffd640 10 monclient(hunting): my global_id is 4149 2024-03-25T18:25:08.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.268+0000 7f1622ffd640 1 -- 172.21.15.16:0/3516323560 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f160c004eb0 con 0x7f162c0b4b50 2024-03-25T18:25:08.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.268+0000 7f1622ffd640 1 -- 172.21.15.16:0/3516323560 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 980585804 0 0) 0x7f162c144b50 con 0x7f162c142dd0 2024-03-25T18:25:08.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.268+0000 7f1622ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:08.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.268+0000 7f1622ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:08.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.268+0000 7f1622ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:08.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.268+0000 7f1622ffd640 10 monclient(hunting): my global_id is 4141 2024-03-25T18:25:08.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.268+0000 7f1622ffd640 1 -- 172.21.15.16:0/3516323560 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f162c145e80 con 0x7f162c142dd0 2024-03-25T18:25:08.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.269+0000 7f1622ffd640 1 -- 172.21.15.16:0/3516323560 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3076023740 0 0) 0x7f162c146310 con 0x7f162c13ff20 2024-03-25T18:25:08.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.269+0000 7f1622ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:08.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.269+0000 7f1622ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:08.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.269+0000 7f1622ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:08.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.269+0000 7f1622ffd640 10 monclient(hunting): my global_id is 4145 2024-03-25T18:25:08.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.269+0000 7f1622ffd640 1 -- 172.21.15.16:0/3516323560 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f162c144b50 con 0x7f162c13ff20 2024-03-25T18:25:08.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.272+0000 7f1622ffd640 1 -- 172.21.15.16:0/3516323560 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1228345811 0 0) 0x7f160c004eb0 con 0x7f162c0b4b50 2024-03-25T18:25:08.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.272+0000 7f1622ffd640 1 -- 172.21.15.16:0/3516323560 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f162c146310 con 0x7f162c0b4b50 2024-03-25T18:25:08.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.272+0000 7f1622ffd640 1 -- 172.21.15.16:0/3516323560 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f161c001a10 con 0x7f162c0b4b50 2024-03-25T18:25:08.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.272+0000 7f1622ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:08.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.272+0000 7f1622ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:08.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.272+0000 7f1622ffd640 10 monclient(hunting): dump: 2024-03-25T18:25:08.275 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:08.275 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:08.275 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:08.275 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:08.275 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:08.275 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:08.275 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:08.275 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:08.275 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:08.275 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:08.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.272+0000 7f1622ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:08.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.273+0000 7f1622ffd640 1 -- 172.21.15.16:0/3516323560 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1668265583 0 0) 0x7f162c145e80 con 0x7f162c142dd0 2024-03-25T18:25:08.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.273+0000 7f1622ffd640 1 -- 172.21.15.16:0/3516323560 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f160c004eb0 con 0x7f162c142dd0 2024-03-25T18:25:08.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.273+0000 7f1622ffd640 1 -- 172.21.15.16:0/3516323560 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f1624001b50 con 0x7f162c142dd0 2024-03-25T18:25:08.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.273+0000 7f1622ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:08.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.273+0000 7f1622ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:08.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.273+0000 7f1622ffd640 10 monclient(hunting): dump: 2024-03-25T18:25:08.276 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:08.276 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:08.276 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:08.276 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:08.276 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:08.276 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:08.276 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:08.276 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:08.276 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:08.276 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:08.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.274+0000 7f1622ffd640 1 -- 172.21.15.16:0/3516323560 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2468454506 0 0) 0x7f162c144b50 con 0x7f162c13ff20 2024-03-25T18:25:08.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.274+0000 7f1622ffd640 1 -- 172.21.15.16:0/3516323560 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f162c145e80 con 0x7f162c13ff20 2024-03-25T18:25:08.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.274+0000 7f1622ffd640 1 -- 172.21.15.16:0/3516323560 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f1614001960 con 0x7f162c13ff20 2024-03-25T18:25:08.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.274+0000 7f1622ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:08.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.274+0000 7f1622ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:08.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.274+0000 7f1622ffd640 10 monclient(hunting): dump: 2024-03-25T18:25:08.278 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:08.278 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:08.278 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:08.279 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:08.279 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:08.279 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:08.279 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:08.279 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:08.279 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:08.279 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:08.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.276+0000 7f1622ffd640 1 -- 172.21.15.16:0/3516323560 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1563686593 0 0) 0x7f162c146310 con 0x7f162c0b4b50 2024-03-25T18:25:08.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.276+0000 7f1622ffd640 1 -- 172.21.15.16:0/3516323560 >> v1:172.21.15.3:6790/0 conn(0x7f162c13ff20 legacy=0x7f162c1dab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:08.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.276+0000 7f1622ffd640 1 -- 172.21.15.16:0/3516323560 >> v1:172.21.15.16:6789/0 conn(0x7f162c142dd0 legacy=0x7f162c1e2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:08.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.276+0000 7f1622ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:08.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.276+0000 7f1622ffd640 1 monclient: found mon.a 2024-03-25T18:25:08.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.276+0000 7f1622ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:08.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.276+0000 7f1622ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:08.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.276+0000 7f1622ffd640 1 -- 172.21.15.16:0/3516323560 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f162c1464a0 con 0x7f162c0b4b50 2024-03-25T18:25:08.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.276+0000 7f1622ffd640 10 monclient: _finish_auth 0 2024-03-25T18:25:08.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.276+0000 7f1622ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:08.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.276+0000 7f1622ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:08.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.276+0000 7f16309d6640 5 monclient: authenticate success, global_id 4149 2024-03-25T18:25:08.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.276+0000 7f16309d6640 10 monclient: _renew_subs 2024-03-25T18:25:08.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.276+0000 7f16309d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:08.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.276+0000 7f16309d6640 1 -- 172.21.15.16:0/3516323560 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f162c137880 con 0x7f162c0b4b50 2024-03-25T18:25:08.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.276+0000 7f16309d6640 10 client.4149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:08.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.276+0000 7f16309d6640 10 monclient: _renew_subs 2024-03-25T18:25:08.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.276+0000 7f16309d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:08.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.276+0000 7f16309d6640 1 -- 172.21.15.16:0/3516323560 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f162c145460 con 0x7f162c0b4b50 2024-03-25T18:25:08.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.276+0000 7f16309d6640 1 librados: init done 2024-03-25T18:25:08.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.277+0000 7f16309d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:08.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.277+0000 7f16309d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:08.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.277+0000 7f16309d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:08.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.277+0000 7f16309d6640 1 -- 172.21.15.16:0/3516323560 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f162c04f730 con 0x7f162c0b4b50 2024-03-25T18:25:08.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.377+0000 7f1622ffd640 1 -- 172.21.15.16:0/3516323560 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f161c001c70 con 0x7f162c0b4b50 2024-03-25T18:25:08.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.377+0000 7f1622ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:08.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.377+0000 7f1622ffd640 1 -- 172.21.15.16:0/3516323560 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f161c003a60 con 0x7f162c0b4b50 2024-03-25T18:25:08.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.377+0000 7f1622ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:08.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.377+0000 7f1622ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:08.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.377+0000 7f1622ffd640 10 monclient: dump: 2024-03-25T18:25:08.379 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:08.379 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:08.379 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:08.379 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:08.379 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:08.379 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:08.379 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:08.379 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:08.379 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:08.379 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:08.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.377+0000 7f1622ffd640 1 -- 172.21.15.16:0/3516323560 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f161c004010 con 0x7f162c0b4b50 2024-03-25T18:25:08.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.377+0000 7f1628d99640 4 set_mon_vals no callback set 2024-03-25T18:25:08.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.378+0000 7f1622ffd640 1 -- 172.21.15.16:0/3516323560 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 279251379 0 0) 0x7f161c006850 con 0x7f162c0b4b50 2024-03-25T18:25:08.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.378+0000 7f1622ffd640 10 client.4149.objecter ms_dispatch 0x7f162c000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T18:25:08.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.378+0000 7f1622ffd640 3 client.4149.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T18:25:08.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.378+0000 7f1622ffd640 3 client.4149.objecter handle_osd_map decoding full epoch 7 2024-03-25T18:25:08.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.378+0000 7f1622ffd640 20 client.4149.objecter dump_active .. 0 homeless 2024-03-25T18:25:08.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.378+0000 7f1622ffd640 1 -- 172.21.15.16:0/3516323560 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f161c006ba0 con 0x7f162c0b4b50 2024-03-25T18:25:08.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.379+0000 7f1622ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:08.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.379+0000 7f1622ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:08.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.452+0000 7f16309d6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9ca8edca-f60e-4490-bc1d-782bc08c5b09", "id": 5}] 2024-03-25T18:25:08.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.452+0000 7f16309d6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9ca8edca-f60e-4490-bc1d-782bc08c5b09", "id": 5}] 2024-03-25T18:25:08.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.452+0000 7f16309d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:08.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.452+0000 7f16309d6640 1 -- 172.21.15.16:0/3516323560 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd new", "uuid": "9ca8edca-f60e-4490-bc1d-782bc08c5b09", "id": 5} v 0) v1 -- 0x7f162c0681c0 con 0x7f162c0b4b50 2024-03-25T18:25:08.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.500+0000 7f1622ffd640 1 -- 172.21.15.16:0/3516323560 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9ca8edca-f60e-4490-bc1d-782bc08c5b09", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 4214150977 0 1988412176) 0x7f161c0062d0 con 0x7f162c0b4b50 2024-03-25T18:25:08.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.500+0000 7f1622ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9ca8edca-f60e-4490-bc1d-782bc08c5b09", "id": 5}] 2024-03-25T18:25:08.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.500+0000 7f1622ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:08.501 INFO:teuthology.orchestra.run.smithi016.stdout:5 2024-03-25T18:25:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.500+0000 7f16309d6640 10 librados: watch_flush enter 2024-03-25T18:25:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.500+0000 7f16309d6640 10 librados: watch_flush exit 2024-03-25T18:25:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.500+0000 7f16309d6640 20 client.4149.objecter shutdown clearing up homeless session... 2024-03-25T18:25:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.500+0000 7f16309d6640 10 client.4149.objecter successfully canceled tick 2024-03-25T18:25:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.500+0000 7f16309d6640 10 monclient: shutdown 2024-03-25T18:25:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.500+0000 7f16309d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.500+0000 7f16309d6640 1 -- 172.21.15.16:0/3516323560 >> v1:172.21.15.3:6789/0 conn(0x7f162c0b4b50 legacy=0x7f162c137110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:08.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.501+0000 7f16309d6640 1 -- 172.21.15.16:0/3516323560 shutdown_connections 2024-03-25T18:25:08.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.501+0000 7f16309d6640 1 -- 172.21.15.16:0/3516323560 >> 172.21.15.16:0/3516323560 conn(0x7f162c066920 msgr2=0x7f162c0a9660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:08.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.501+0000 7f16309d6640 1 -- 172.21.15.16:0/3516323560 shutdown_connections 2024-03-25T18:25:08.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.501+0000 7f16309d6640 1 -- 172.21.15.16:0/3516323560 wait complete. 2024-03-25T18:25:08.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.501+0000 7f16309d6640 1 librados: shutdown 2024-03-25T18:25:08.512 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd new 742ad6e9-6a86-4fb6-bd82-020c991fba92 6 2024-03-25T18:25:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.611+0000 7f9209572640 10 monclient: get_monmap_and_config 2024-03-25T18:25:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.611+0000 7f9209572640 10 monclient: build_initial_monmap 2024-03-25T18:25:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.611+0000 7f9209572640 10 monclient: monmap: 2024-03-25T18:25:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 0 2024-03-25T18:25:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:25:08.612833+0000 2024-03-25T18:25:08.613 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:25:08.612833+0000 2024-03-25T18:25:08.613 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:08.613 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:08.613 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:08.613 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:25:08.613 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:25:08.613 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:08.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.612+0000 7f9209572640 1 Processor -- start 2024-03-25T18:25:08.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.612+0000 7f9209572640 1 -- start start 2024-03-25T18:25:08.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.612+0000 7f9209572640 10 monclient: init 2024-03-25T18:25:08.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.613+0000 7f9209572640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:08.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.613+0000 7f9209572640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:25:08.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.613+0000 7f9209572640 10 monclient(hunting): picked mon.noname-c con 0x7f92040b4b50 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:08.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.613+0000 7f9209572640 10 monclient(hunting): picked mon.noname-a con 0x7f920413ff20 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:08.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.613+0000 7f9209572640 10 monclient(hunting): picked mon.noname-b con 0x7f9204142dd0 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:08.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.613+0000 7f9209572640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92040ab440 con 0x7f920413ff20 2024-03-25T18:25:08.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.613+0000 7f9209572640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92040ab6b0 con 0x7f92040b4b50 2024-03-25T18:25:08.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.613+0000 7f9209572640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92040a9f10 con 0x7f9204142dd0 2024-03-25T18:25:08.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.613+0000 7f9209572640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:08.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.613+0000 7f9209572640 10 monclient(hunting): authenticate will time out at 7365.748675s 2024-03-25T18:25:08.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.618+0000 7f92037fe640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f9204142dd0 0x7f920414df50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.16:48926/0 (socket says 172.21.15.16:48926) 2024-03-25T18:25:08.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.618+0000 7f92037fe640 1 -- 172.21.15.16:0/724149527 learned_addr learned my addr 172.21.15.16:0/724149527 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-25T18:25:08.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.623+0000 7f92027fc640 1 -- 172.21.15.16:0/724149527 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1441781203 0 0) 0x7f92040a9f10 con 0x7f9204142dd0 2024-03-25T18:25:08.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.623+0000 7f92027fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:08.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.623+0000 7f92027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:08.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.623+0000 7f92027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:08.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.623+0000 7f92027fc640 10 monclient(hunting): my global_id is 4148 2024-03-25T18:25:08.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.623+0000 7f92027fc640 1 -- 172.21.15.16:0/724149527 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91f0003520 con 0x7f9204142dd0 2024-03-25T18:25:08.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.624+0000 7f92027fc640 1 -- 172.21.15.16:0/724149527 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2416639890 0 0) 0x7f92040ab440 con 0x7f920413ff20 2024-03-25T18:25:08.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.624+0000 7f92027fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:08.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.624+0000 7f92027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:08.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.624+0000 7f92027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:08.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.624+0000 7f92027fc640 10 monclient(hunting): my global_id is 4152 2024-03-25T18:25:08.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.624+0000 7f92027fc640 1 -- 172.21.15.16:0/724149527 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92040a9f10 con 0x7f920413ff20 2024-03-25T18:25:08.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.627+0000 7f92027fc640 1 -- 172.21.15.16:0/724149527 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1182300459 0 0) 0x7f91f0003520 con 0x7f9204142dd0 2024-03-25T18:25:08.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.627+0000 7f92027fc640 1 -- 172.21.15.16:0/724149527 >> v1:172.21.15.16:6789/0 conn(0x7f92040b4b50 legacy=0x7f920413f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:08.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.627+0000 7f92027fc640 1 -- 172.21.15.16:0/724149527 >> v1:172.21.15.3:6789/0 conn(0x7f920413ff20 legacy=0x7f92041426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:08.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.627+0000 7f92027fc640 10 monclient: _finish_hunting 0 2024-03-25T18:25:08.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.627+0000 7f92027fc640 1 monclient: found mon.noname-b 2024-03-25T18:25:08.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.627+0000 7f92027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:08.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.627+0000 7f92027fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-25T18:25:08.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.627+0000 7f92027fc640 1 -- 172.21.15.16:0/724149527 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92040aa0a0 con 0x7f9204142dd0 2024-03-25T18:25:08.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.627+0000 7f92027fc640 10 monclient: _finish_auth 0 2024-03-25T18:25:08.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.627+0000 7f92027fc640 10 monclient: _check_auth_tickets 2024-03-25T18:25:08.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.627+0000 7f92027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:08.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.627+0000 7f92027fc640 1 -- 172.21.15.16:0/724149527 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f91fc001a10 con 0x7f9204142dd0 2024-03-25T18:25:08.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.627+0000 7f9209572640 5 monclient: authenticate success, global_id 4148 2024-03-25T18:25:08.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.627+0000 7f9209572640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:08.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.627+0000 7f92027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:08.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.627+0000 7f92027fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:08.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.627+0000 7f92027fc640 10 monclient: dump: 2024-03-25T18:25:08.630 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:08.630 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:08.630 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:08.630 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:08.630 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:08.630 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:08.630 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:08.630 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:08.630 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:08.630 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:08.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.627+0000 7f9209572640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:08.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.654+0000 7f92027fc640 1 -- 172.21.15.16:0/724149527 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91fc001b70 con 0x7f9204142dd0 2024-03-25T18:25:08.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.655+0000 7f92027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:08.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.655+0000 7f92027fc640 1 -- 172.21.15.16:0/724149527 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f91fc001f00 con 0x7f9204142dd0 2024-03-25T18:25:08.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.655+0000 7f92027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:08.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.655+0000 7f92027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:08.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.655+0000 7f92027fc640 10 monclient: dump: 2024-03-25T18:25:08.656 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:08.656 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:08.656 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:08.656 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:08.656 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:08.656 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:08.656 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:08.656 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:08.656 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:08.656 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:08.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.655+0000 7f9209572640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:08.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.655+0000 7f9209572640 4 set_mon_vals no callback set 2024-03-25T18:25:08.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.655+0000 7f9209572640 10 monclient: shutdown 2024-03-25T18:25:08.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.655+0000 7f9209572640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:08.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.655+0000 7f9209572640 1 -- 172.21.15.16:0/724149527 >> v1:172.21.15.3:6790/0 conn(0x7f9204142dd0 legacy=0x7f920414df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:08.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.655+0000 7f9209572640 1 -- 172.21.15.16:0/724149527 shutdown_connections 2024-03-25T18:25:08.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.655+0000 7f9209572640 1 -- 172.21.15.16:0/724149527 >> 172.21.15.16:0/724149527 conn(0x7f9204066920 msgr2=0x7f9204066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:08.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.655+0000 7f9209572640 1 -- 172.21.15.16:0/724149527 shutdown_connections 2024-03-25T18:25:08.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.655+0000 7f9209572640 1 -- 172.21.15.16:0/724149527 wait complete. 2024-03-25T18:25:08.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.656+0000 7f9209572640 10 monclient: build_initial_monmap 2024-03-25T18:25:08.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.656+0000 7f9209572640 10 monclient: monmap: 2024-03-25T18:25:08.659 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 0 2024-03-25T18:25:08.659 INFO:teuthology.orchestra.run.smithi016.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:08.659 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 0.000000 2024-03-25T18:25:08.659 INFO:teuthology.orchestra.run.smithi016.stderr:created 0.000000 2024-03-25T18:25:08.659 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:08.660 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 1 2024-03-25T18:25:08.660 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:08.660 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:25:08.660 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:25:08.660 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:08.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.656+0000 7f9209572640 1 librados: starting msgr at 2024-03-25T18:25:08.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.656+0000 7f9209572640 1 librados: starting objecter 2024-03-25T18:25:08.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.656+0000 7f9209572640 1 Processor -- start 2024-03-25T18:25:08.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.656+0000 7f9209572640 1 -- start start 2024-03-25T18:25:08.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.656+0000 7f9209572640 1 librados: setting wanted keys 2024-03-25T18:25:08.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.656+0000 7f9209572640 1 librados: calling monclient init 2024-03-25T18:25:08.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.656+0000 7f9209572640 10 monclient: init 2024-03-25T18:25:08.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.657+0000 7f9209572640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:08.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.657+0000 7f9209572640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:08.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.657+0000 7f9209572640 10 monclient(hunting): picked mon.noname-c con 0x7f92040b4b50 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:08.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.657+0000 7f9209572640 10 monclient(hunting): picked mon.noname-b con 0x7f920413ff20 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:08.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.657+0000 7f9209572640 10 monclient(hunting): picked mon.noname-a con 0x7f9204142dd0 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:08.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.657+0000 7f9209572640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9204134d70 con 0x7f9204142dd0 2024-03-25T18:25:08.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.657+0000 7f9209572640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92041446a0 con 0x7f920413ff20 2024-03-25T18:25:08.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.657+0000 7f9209572640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9204144980 con 0x7f92040b4b50 2024-03-25T18:25:08.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.657+0000 7f9209572640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:08.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.657+0000 7f9209572640 10 monclient(hunting): authenticate will time out at 7365.792951s 2024-03-25T18:25:08.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.657+0000 7f92037fe640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f9204142dd0 0x7f9204134660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.16:45572/0 (socket says 172.21.15.16:45572) 2024-03-25T18:25:08.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.657+0000 7f92037fe640 1 -- 172.21.15.16:0/713256802 learned_addr learned my addr 172.21.15.16:0/713256802 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-25T18:25:08.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.658+0000 7f91fb7fe640 10 client.?.objecter ms_handle_connect 0x7f920413ff20 2024-03-25T18:25:08.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.658+0000 7f91fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:08.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.659+0000 7f91fb7fe640 10 client.?.objecter ms_handle_connect 0x7f9204142dd0 2024-03-25T18:25:08.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.659+0000 7f91fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:08.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.660+0000 7f91fb7fe640 1 -- 172.21.15.16:0/713256802 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156711313 0 0) 0x7f92041446a0 con 0x7f920413ff20 2024-03-25T18:25:08.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.660+0000 7f91fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:08.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.660+0000 7f91fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:08.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.660+0000 7f91fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:08.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.660+0000 7f91fb7fe640 10 monclient(hunting): my global_id is 4147 2024-03-25T18:25:08.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.660+0000 7f91fb7fe640 1 -- 172.21.15.16:0/713256802 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91e4004eb0 con 0x7f920413ff20 2024-03-25T18:25:08.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.660+0000 7f91fb7fe640 10 client.?.objecter ms_handle_connect 0x7f92040b4b50 2024-03-25T18:25:08.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.660+0000 7f91fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:08.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.662+0000 7f91fb7fe640 1 -- 172.21.15.16:0/713256802 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3384007943 0 0) 0x7f9204134d70 con 0x7f9204142dd0 2024-03-25T18:25:08.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.662+0000 7f91fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:08.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.662+0000 7f91fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:08.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.662+0000 7f91fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:08.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.662+0000 7f91fb7fe640 10 monclient(hunting): my global_id is 4155 2024-03-25T18:25:08.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.662+0000 7f91fb7fe640 1 -- 172.21.15.16:0/713256802 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92041446a0 con 0x7f9204142dd0 2024-03-25T18:25:08.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.662+0000 7f91fb7fe640 1 -- 172.21.15.16:0/713256802 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2430637453 0 0) 0x7f9204144980 con 0x7f92040b4b50 2024-03-25T18:25:08.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.662+0000 7f91fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:08.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.662+0000 7f91fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:08.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.662+0000 7f91fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:08.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.662+0000 7f91fb7fe640 10 monclient(hunting): my global_id is 4151 2024-03-25T18:25:08.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.662+0000 7f91fb7fe640 1 -- 172.21.15.16:0/713256802 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9204134d70 con 0x7f92040b4b50 2024-03-25T18:25:08.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.665+0000 7f91fb7fe640 1 -- 172.21.15.16:0/713256802 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1030542871 0 0) 0x7f91e4004eb0 con 0x7f920413ff20 2024-03-25T18:25:08.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.665+0000 7f91fb7fe640 1 -- 172.21.15.16:0/713256802 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9204144980 con 0x7f920413ff20 2024-03-25T18:25:08.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.665+0000 7f91fb7fe640 1 -- 172.21.15.16:0/713256802 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f91ec001a10 con 0x7f920413ff20 2024-03-25T18:25:08.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.665+0000 7f91fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:08.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.665+0000 7f91fb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:08.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.665+0000 7f91fb7fe640 10 monclient(hunting): dump: 2024-03-25T18:25:08.667 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:08.667 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:08.667 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:08.667 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:08.667 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:08.667 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:08.667 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:08.667 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:08.667 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:08.667 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:08.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.665+0000 7f91fb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.669+0000 7f91fb7fe640 1 -- 172.21.15.16:0/713256802 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2619014033 0 0) 0x7f9204134d70 con 0x7f92040b4b50 2024-03-25T18:25:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.669+0000 7f91fb7fe640 1 -- 172.21.15.16:0/713256802 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91e4004eb0 con 0x7f92040b4b50 2024-03-25T18:25:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.669+0000 7f91fb7fe640 1 -- 172.21.15.16:0/713256802 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f91f40017f0 con 0x7f92040b4b50 2024-03-25T18:25:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.669+0000 7f91fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.669+0000 7f91fb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.669+0000 7f91fb7fe640 10 monclient(hunting): dump: 2024-03-25T18:25:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:08.672 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.669+0000 7f91fb7fe640 1 -- 172.21.15.16:0/713256802 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 619028931 0 0) 0x7f9204144980 con 0x7f920413ff20 2024-03-25T18:25:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.669+0000 7f91fb7fe640 1 -- 172.21.15.16:0/713256802 >> v1:172.21.15.3:6790/0 conn(0x7f92040b4b50 legacy=0x7f920413a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.669+0000 7f91fb7fe640 1 -- 172.21.15.16:0/713256802 >> v1:172.21.15.3:6789/0 conn(0x7f9204142dd0 legacy=0x7f9204134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.669+0000 7f91fb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.669+0000 7f91fb7fe640 1 monclient: found mon.b 2024-03-25T18:25:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.669+0000 7f91fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.669+0000 7f91fb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.669+0000 7f91fb7fe640 1 -- 172.21.15.16:0/713256802 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f920413ae70 con 0x7f920413ff20 2024-03-25T18:25:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.669+0000 7f91fb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.669+0000 7f91fb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.669+0000 7f91fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.669+0000 7f9209572640 5 monclient: authenticate success, global_id 4147 2024-03-25T18:25:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.669+0000 7f9209572640 10 monclient: _renew_subs 2024-03-25T18:25:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.669+0000 7f9209572640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.669+0000 7f9209572640 1 -- 172.21.15.16:0/713256802 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9204144da0 con 0x7f920413ff20 2024-03-25T18:25:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.669+0000 7f9209572640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.669+0000 7f9209572640 10 monclient: _renew_subs 2024-03-25T18:25:08.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.669+0000 7f9209572640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:08.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.669+0000 7f9209572640 1 -- 172.21.15.16:0/713256802 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9204139fa0 con 0x7f920413ff20 2024-03-25T18:25:08.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.669+0000 7f9209572640 1 librados: init done 2024-03-25T18:25:08.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.670+0000 7f9209572640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:08.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.670+0000 7f9209572640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:08.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.670+0000 7f9209572640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:08.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.670+0000 7f9209572640 1 -- 172.21.15.16:0/713256802 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f920404f730 con 0x7f920413ff20 2024-03-25T18:25:08.771 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-25T18:25:08.770+0000 7f0c86a58040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:25:08.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.782+0000 7f91fb7fe640 1 -- 172.21.15.16:0/713256802 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91ec003690 con 0x7f920413ff20 2024-03-25T18:25:08.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.782+0000 7f91fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:08.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.782+0000 7f91fb7fe640 1 -- 172.21.15.16:0/713256802 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f91ec0039a0 con 0x7f920413ff20 2024-03-25T18:25:08.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.782+0000 7f91fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:08.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.782+0000 7f91fb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:08.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.782+0000 7f91fb7fe640 10 monclient: dump: 2024-03-25T18:25:08.784 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:08.784 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:08.784 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:08.785 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.782+0000 7f91fb7fe640 1 -- 172.21.15.16:0/713256802 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f91ec003e90 con 0x7f920413ff20 2024-03-25T18:25:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.782+0000 7f9201ffb640 4 set_mon_vals no callback set 2024-03-25T18:25:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.782+0000 7f91fb7fe640 1 -- 172.21.15.16:0/713256802 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 1823025032 0 0) 0x7f91ec0056a0 con 0x7f920413ff20 2024-03-25T18:25:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.782+0000 7f91fb7fe640 10 client.4147.objecter ms_dispatch 0x7f9204000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T18:25:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.782+0000 7f91fb7fe640 3 client.4147.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T18:25:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.782+0000 7f91fb7fe640 3 client.4147.objecter handle_osd_map decoding full epoch 8 2024-03-25T18:25:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.783+0000 7f91fb7fe640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-25T18:25:08.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.783+0000 7f91fb7fe640 1 -- 172.21.15.16:0/713256802 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f91ec005a20 con 0x7f920413ff20 2024-03-25T18:25:08.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.783+0000 7f91fb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:08.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.783+0000 7f91fb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:08.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.857+0000 7f9209572640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "742ad6e9-6a86-4fb6-bd82-020c991fba92", "id": 6}] 2024-03-25T18:25:08.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.857+0000 7f9209572640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "742ad6e9-6a86-4fb6-bd82-020c991fba92", "id": 6}] 2024-03-25T18:25:08.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.857+0000 7f9209572640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:08.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.857+0000 7f9209572640 1 -- 172.21.15.16:0/713256802 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd new", "uuid": "742ad6e9-6a86-4fb6-bd82-020c991fba92", "id": 6} v 0) v1 -- 0x7f92040681c0 con 0x7f920413ff20 2024-03-25T18:25:08.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.919+0000 7f91fb7fe640 1 -- 172.21.15.16:0/713256802 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "742ad6e9-6a86-4fb6-bd82-020c991fba92", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 1293656941 0 1113811849) 0x7f91ec005120 con 0x7f920413ff20 2024-03-25T18:25:08.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.919+0000 7f91fb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "742ad6e9-6a86-4fb6-bd82-020c991fba92", "id": 6}] 2024-03-25T18:25:08.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.919+0000 7f91fb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:08.921 INFO:teuthology.orchestra.run.smithi016.stdout:6 2024-03-25T18:25:08.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.920+0000 7f9209572640 10 librados: watch_flush enter 2024-03-25T18:25:08.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.920+0000 7f9209572640 10 librados: watch_flush exit 2024-03-25T18:25:08.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.920+0000 7f9209572640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-25T18:25:08.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.920+0000 7f9209572640 10 client.4147.objecter successfully canceled tick 2024-03-25T18:25:08.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.920+0000 7f9209572640 10 monclient: shutdown 2024-03-25T18:25:08.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.920+0000 7f9209572640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:08.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.920+0000 7f9209572640 1 -- 172.21.15.16:0/713256802 >> v1:172.21.15.16:6789/0 conn(0x7f920413ff20 legacy=0x7f9204133f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:08.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.920+0000 7f9209572640 1 -- 172.21.15.16:0/713256802 shutdown_connections 2024-03-25T18:25:08.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.920+0000 7f9209572640 1 -- 172.21.15.16:0/713256802 >> 172.21.15.16:0/713256802 conn(0x7f9204066920 msgr2=0x7f9204066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:08.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.921+0000 7f9209572640 1 -- 172.21.15.16:0/713256802 shutdown_connections 2024-03-25T18:25:08.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.921+0000 7f9209572640 1 -- 172.21.15.16:0/713256802 wait complete. 2024-03-25T18:25:08.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:08.921+0000 7f9209572640 1 librados: shutdown 2024-03-25T18:25:08.932 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd new 98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c 7 2024-03-25T18:25:08.940 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-03-25T18:25:08.939+0000 7f49c1ff1040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:25:09.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.031+0000 7f749144a640 10 monclient: get_monmap_and_config 2024-03-25T18:25:09.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.031+0000 7f749144a640 10 monclient: build_initial_monmap 2024-03-25T18:25:09.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.031+0000 7f749144a640 10 monclient: monmap: 2024-03-25T18:25:09.032 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 0 2024-03-25T18:25:09.032 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:09.032 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:25:09.032649+0000 2024-03-25T18:25:09.032 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:25:09.032649+0000 2024-03-25T18:25:09.032 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:09.033 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:09.033 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:09.033 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:25:09.033 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:25:09.033 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:09.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.032+0000 7f749144a640 1 Processor -- start 2024-03-25T18:25:09.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.032+0000 7f749144a640 1 -- start start 2024-03-25T18:25:09.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.032+0000 7f749144a640 10 monclient: init 2024-03-25T18:25:09.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.032+0000 7f749144a640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:09.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.032+0000 7f749144a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:09.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.032+0000 7f749144a640 10 monclient(hunting): picked mon.noname-b con 0x7f748c0b4f00 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:09.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.032+0000 7f749144a640 10 monclient(hunting): picked mon.noname-a con 0x7f748c0b4510 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:09.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.032+0000 7f749144a640 10 monclient(hunting): picked mon.noname-c con 0x7f748c142ad0 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:09.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.032+0000 7f749144a640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f748c0a9d90 con 0x7f748c0b4510 2024-03-25T18:25:09.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.032+0000 7f749144a640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f748c0aa2b0 con 0x7f748c142ad0 2024-03-25T18:25:09.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.032+0000 7f749144a640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f748c13fc60 con 0x7f748c0b4f00 2024-03-25T18:25:09.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.032+0000 7f749144a640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:09.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.032+0000 7f749144a640 10 monclient(hunting): authenticate will time out at 7366.168543s 2024-03-25T18:25:09.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.033+0000 7f748a7fc640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f748c0b4510 0x7f748c13f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.16:45580/0 (socket says 172.21.15.16:45580) 2024-03-25T18:25:09.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.033+0000 7f748a7fc640 1 -- 172.21.15.16:0/288450707 learned_addr learned my addr 172.21.15.16:0/288450707 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-25T18:25:09.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.035+0000 7f7489ffb640 1 -- 172.21.15.16:0/288450707 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873626953 0 0) 0x7f748c0aa2b0 con 0x7f748c142ad0 2024-03-25T18:25:09.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.035+0000 7f7489ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:09.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.035+0000 7f7489ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:09.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.035+0000 7f7489ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:09.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.035+0000 7f7489ffb640 10 monclient(hunting): my global_id is 4150 2024-03-25T18:25:09.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.035+0000 7f7489ffb640 1 -- 172.21.15.16:0/288450707 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7474003520 con 0x7f748c142ad0 2024-03-25T18:25:09.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.035+0000 7f7489ffb640 1 -- 172.21.15.16:0/288450707 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1576970300 0 0) 0x7f748c13fc60 con 0x7f748c0b4f00 2024-03-25T18:25:09.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.035+0000 7f7489ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:09.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.035+0000 7f7489ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:09.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.035+0000 7f7489ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:09.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.035+0000 7f7489ffb640 10 monclient(hunting): my global_id is 4154 2024-03-25T18:25:09.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.035+0000 7f7489ffb640 1 -- 172.21.15.16:0/288450707 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f748c0aa2b0 con 0x7f748c0b4f00 2024-03-25T18:25:09.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.036+0000 7f7489ffb640 1 -- 172.21.15.16:0/288450707 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 747229792 0 0) 0x7f748c0a9d90 con 0x7f748c0b4510 2024-03-25T18:25:09.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.036+0000 7f7489ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:09.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.036+0000 7f7489ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:09.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.036+0000 7f7489ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:09.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.036+0000 7f7489ffb640 10 monclient(hunting): my global_id is 4158 2024-03-25T18:25:09.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.036+0000 7f7489ffb640 1 -- 172.21.15.16:0/288450707 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f748c13fc60 con 0x7f748c0b4510 2024-03-25T18:25:09.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.039+0000 7f7489ffb640 1 -- 172.21.15.16:0/288450707 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2322692701 0 0) 0x7f7474003520 con 0x7f748c142ad0 2024-03-25T18:25:09.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.039+0000 7f7489ffb640 1 -- 172.21.15.16:0/288450707 >> v1:172.21.15.3:6790/0 conn(0x7f748c0b4f00 legacy=0x7f748c1423c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:09.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.039+0000 7f7489ffb640 1 -- 172.21.15.16:0/288450707 >> v1:172.21.15.3:6789/0 conn(0x7f748c0b4510 legacy=0x7f748c13f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:09.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.039+0000 7f7489ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:25:09.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.039+0000 7f7489ffb640 1 monclient: found mon.noname-c 2024-03-25T18:25:09.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.039+0000 7f7489ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:09.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.039+0000 7f7489ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:25:09.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.039+0000 7f7489ffb640 1 -- 172.21.15.16:0/288450707 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f748c13fdf0 con 0x7f748c142ad0 2024-03-25T18:25:09.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.039+0000 7f7489ffb640 10 monclient: _finish_auth 0 2024-03-25T18:25:09.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.039+0000 7f7489ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:25:09.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.039+0000 7f7489ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:09.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.039+0000 7f7489ffb640 1 -- 172.21.15.16:0/288450707 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7484001a10 con 0x7f748c142ad0 2024-03-25T18:25:09.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.039+0000 7f7489ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:09.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.039+0000 7f7489ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:09.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.039+0000 7f7489ffb640 10 monclient: dump: 2024-03-25T18:25:09.041 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:09.041 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:09.041 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:09.041 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:09.041 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:09.041 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:09.041 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:09.041 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:09.041 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:09.041 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:09.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.039+0000 7f749144a640 5 monclient: authenticate success, global_id 4150 2024-03-25T18:25:09.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.039+0000 7f749144a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:09.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.040+0000 7f7489ffb640 1 -- 172.21.15.16:0/288450707 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7484001b70 con 0x7f748c142ad0 2024-03-25T18:25:09.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.040+0000 7f7489ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:09.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.040+0000 7f7489ffb640 1 -- 172.21.15.16:0/288450707 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7484001f80 con 0x7f748c142ad0 2024-03-25T18:25:09.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.040+0000 7f7489ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:09.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.040+0000 7f7489ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:09.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.040+0000 7f7489ffb640 10 monclient: dump: 2024-03-25T18:25:09.043 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:09.043 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:09.043 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:09.043 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:09.043 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:09.043 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:09.043 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:09.043 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:09.043 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:09.044 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:09.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.040+0000 7f749144a640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:09.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.040+0000 7f749144a640 4 set_mon_vals no callback set 2024-03-25T18:25:09.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.041+0000 7f749144a640 10 monclient: shutdown 2024-03-25T18:25:09.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.041+0000 7f749144a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:09.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.041+0000 7f749144a640 1 -- 172.21.15.16:0/288450707 >> v1:172.21.15.16:6789/0 conn(0x7f748c142ad0 legacy=0x7f748c14dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:09.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.041+0000 7f749144a640 1 -- 172.21.15.16:0/288450707 shutdown_connections 2024-03-25T18:25:09.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.041+0000 7f749144a640 1 -- 172.21.15.16:0/288450707 >> 172.21.15.16:0/288450707 conn(0x7f748c066b00 msgr2=0x7f748c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:09.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.041+0000 7f749144a640 1 -- 172.21.15.16:0/288450707 shutdown_connections 2024-03-25T18:25:09.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.042+0000 7f749144a640 1 -- 172.21.15.16:0/288450707 wait complete. 2024-03-25T18:25:09.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.042+0000 7f749144a640 10 monclient: build_initial_monmap 2024-03-25T18:25:09.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.042+0000 7f749144a640 10 monclient: monmap: 2024-03-25T18:25:09.046 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 0 2024-03-25T18:25:09.047 INFO:teuthology.orchestra.run.smithi016.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:09.047 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 0.000000 2024-03-25T18:25:09.047 INFO:teuthology.orchestra.run.smithi016.stderr:created 0.000000 2024-03-25T18:25:09.047 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:09.047 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 1 2024-03-25T18:25:09.047 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:09.047 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:25:09.047 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:25:09.047 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:09.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.043+0000 7f749144a640 1 librados: starting msgr at 2024-03-25T18:25:09.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.043+0000 7f749144a640 1 librados: starting objecter 2024-03-25T18:25:09.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.043+0000 7f749144a640 1 Processor -- start 2024-03-25T18:25:09.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.043+0000 7f749144a640 1 -- start start 2024-03-25T18:25:09.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.043+0000 7f749144a640 1 librados: setting wanted keys 2024-03-25T18:25:09.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.043+0000 7f749144a640 1 librados: calling monclient init 2024-03-25T18:25:09.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.043+0000 7f749144a640 10 monclient: init 2024-03-25T18:25:09.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.044+0000 7f749144a640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:09.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.044+0000 7f749144a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:09.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.044+0000 7f749144a640 10 monclient(hunting): picked mon.noname-a con 0x7f748c0b4510 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:09.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.044+0000 7f749144a640 10 monclient(hunting): picked mon.noname-b con 0x7f748c0b4f00 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:09.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.044+0000 7f749144a640 10 monclient(hunting): picked mon.noname-c con 0x7f748c142ad0 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:09.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.044+0000 7f749144a640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f748c146640 con 0x7f748c0b4510 2024-03-25T18:25:09.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.044+0000 7f749144a640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f748c0b7880 con 0x7f748c0b4f00 2024-03-25T18:25:09.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.044+0000 7f749144a640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f748c0b7b60 con 0x7f748c142ad0 2024-03-25T18:25:09.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.044+0000 7f749144a640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:09.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.044+0000 7f749144a640 10 monclient(hunting): authenticate will time out at 7366.180266s 2024-03-25T18:25:09.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.045+0000 7f748affd640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f748c0b4510 0x7f748c0b8340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.16:45588/0 (socket says 172.21.15.16:45588) 2024-03-25T18:25:09.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.045+0000 7f748affd640 1 -- 172.21.15.16:0/3509290817 learned_addr learned my addr 172.21.15.16:0/3509290817 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-25T18:25:09.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.045+0000 7f74737fe640 10 client.?.objecter ms_handle_connect 0x7f748c0b4510 2024-03-25T18:25:09.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.045+0000 7f74737fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:09.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.045+0000 7f74737fe640 10 client.?.objecter ms_handle_connect 0x7f748c0b4f00 2024-03-25T18:25:09.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.045+0000 7f74737fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:09.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.045+0000 7f74737fe640 10 client.?.objecter ms_handle_connect 0x7f748c142ad0 2024-03-25T18:25:09.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.045+0000 7f74737fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:09.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.047+0000 7f74737fe640 1 -- 172.21.15.16:0/3509290817 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3664143131 0 0) 0x7f748c0b7880 con 0x7f748c0b4f00 2024-03-25T18:25:09.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.047+0000 7f74737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:09.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.047+0000 7f74737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:09.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.047+0000 7f74737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:09.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.047+0000 7f74737fe640 10 monclient(hunting): my global_id is 4153 2024-03-25T18:25:09.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.047+0000 7f74737fe640 1 -- 172.21.15.16:0/3509290817 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7478004eb0 con 0x7f748c0b4f00 2024-03-25T18:25:09.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.047+0000 7f74737fe640 1 -- 172.21.15.16:0/3509290817 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4162399948 0 0) 0x7f748c0b7b60 con 0x7f748c142ad0 2024-03-25T18:25:09.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.047+0000 7f74737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:09.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.047+0000 7f74737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:09.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.047+0000 7f74737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:09.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.047+0000 7f74737fe640 10 monclient(hunting): my global_id is 4157 2024-03-25T18:25:09.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.047+0000 7f74737fe640 1 -- 172.21.15.16:0/3509290817 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f748c0b7880 con 0x7f748c142ad0 2024-03-25T18:25:09.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.047+0000 7f74737fe640 1 -- 172.21.15.16:0/3509290817 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3769497761 0 0) 0x7f748c146640 con 0x7f748c0b4510 2024-03-25T18:25:09.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.047+0000 7f74737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:09.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.047+0000 7f74737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:09.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.047+0000 7f74737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:09.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.047+0000 7f74737fe640 10 monclient(hunting): my global_id is 4161 2024-03-25T18:25:09.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.047+0000 7f74737fe640 1 -- 172.21.15.16:0/3509290817 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f748c0b7b60 con 0x7f748c0b4510 2024-03-25T18:25:09.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.052+0000 7f74737fe640 1 -- 172.21.15.16:0/3509290817 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2516015851 0 0) 0x7f7478004eb0 con 0x7f748c0b4f00 2024-03-25T18:25:09.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.052+0000 7f74737fe640 1 -- 172.21.15.16:0/3509290817 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f748c146640 con 0x7f748c0b4f00 2024-03-25T18:25:09.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.052+0000 7f74737fe640 1 -- 172.21.15.16:0/3509290817 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f74800018f0 con 0x7f748c0b4f00 2024-03-25T18:25:09.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.052+0000 7f74737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:09.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.052+0000 7f74737fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:09.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.052+0000 7f74737fe640 10 monclient(hunting): dump: 2024-03-25T18:25:09.054 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:09.054 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:09.054 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:09.054 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:09.054 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:09.054 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:09.054 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:09.054 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:09.054 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:09.054 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:09.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.052+0000 7f74737fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:09.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.052+0000 7f74737fe640 1 -- 172.21.15.16:0/3509290817 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1594861186 0 0) 0x7f748c0b7880 con 0x7f748c142ad0 2024-03-25T18:25:09.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.052+0000 7f74737fe640 1 -- 172.21.15.16:0/3509290817 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7478004eb0 con 0x7f748c142ad0 2024-03-25T18:25:09.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.052+0000 7f74737fe640 1 -- 172.21.15.16:0/3509290817 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7484001e80 con 0x7f748c142ad0 2024-03-25T18:25:09.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.052+0000 7f74737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:09.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.052+0000 7f74737fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:09.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.052+0000 7f74737fe640 10 monclient(hunting): dump: 2024-03-25T18:25:09.055 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:09.055 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:09.055 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:09.055 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:09.055 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:09.055 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:09.055 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:09.055 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:09.055 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:09.055 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:09.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.053+0000 7f74737fe640 1 -- 172.21.15.16:0/3509290817 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1163693948 0 0) 0x7f748c0b7b60 con 0x7f748c0b4510 2024-03-25T18:25:09.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.053+0000 7f74737fe640 1 -- 172.21.15.16:0/3509290817 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f748c0b7880 con 0x7f748c0b4510 2024-03-25T18:25:09.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.053+0000 7f74737fe640 1 -- 172.21.15.16:0/3509290817 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f747c001930 con 0x7f748c0b4510 2024-03-25T18:25:09.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.053+0000 7f74737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:09.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.053+0000 7f74737fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:09.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.053+0000 7f74737fe640 10 monclient(hunting): dump: 2024-03-25T18:25:09.057 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:09.057 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:09.057 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:09.057 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:09.057 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:09.057 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:09.057 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:09.057 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:09.057 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:09.058 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:09.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.055+0000 7f74737fe640 1 -- 172.21.15.16:0/3509290817 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 628200165 0 0) 0x7f748c146640 con 0x7f748c0b4f00 2024-03-25T18:25:09.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.055+0000 7f74737fe640 1 -- 172.21.15.16:0/3509290817 >> v1:172.21.15.3:6790/0 conn(0x7f748c142ad0 legacy=0x7f748c1e67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:09.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.055+0000 7f74737fe640 1 -- 172.21.15.16:0/3509290817 >> v1:172.21.15.3:6789/0 conn(0x7f748c0b4510 legacy=0x7f748c0b8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:09.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.055+0000 7f74737fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:09.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.055+0000 7f74737fe640 1 monclient: found mon.b 2024-03-25T18:25:09.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.055+0000 7f74737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:09.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.055+0000 7f74737fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:09.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.055+0000 7f74737fe640 1 -- 172.21.15.16:0/3509290817 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f748c0b5ad0 con 0x7f748c0b4f00 2024-03-25T18:25:09.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.055+0000 7f74737fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:09.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.055+0000 7f74737fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:09.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.055+0000 7f74737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:09.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.055+0000 7f749144a640 5 monclient: authenticate success, global_id 4153 2024-03-25T18:25:09.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.055+0000 7f749144a640 10 monclient: _renew_subs 2024-03-25T18:25:09.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.055+0000 7f749144a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:09.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.055+0000 7f749144a640 1 -- 172.21.15.16:0/3509290817 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f748c071f30 con 0x7f748c0b4f00 2024-03-25T18:25:09.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.055+0000 7f749144a640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:09.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.055+0000 7f749144a640 10 monclient: _renew_subs 2024-03-25T18:25:09.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.055+0000 7f749144a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:09.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.055+0000 7f749144a640 1 -- 172.21.15.16:0/3509290817 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f748c0b8c30 con 0x7f748c0b4f00 2024-03-25T18:25:09.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.055+0000 7f749144a640 1 librados: init done 2024-03-25T18:25:09.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.056+0000 7f749144a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:09.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.056+0000 7f749144a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:09.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.056+0000 7f749144a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:09.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.056+0000 7f749144a640 1 -- 172.21.15.16:0/3509290817 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f748c0b8080 con 0x7f748c0b4f00 2024-03-25T18:25:09.069 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-25T18:25:09.067+0000 7f0c86a58040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:25:09.122 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-25T18:25:09.121+0000 7f0c86a58040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:25:09.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.158+0000 7f74737fe640 1 -- 172.21.15.16:0/3509290817 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7480001d40 con 0x7f748c0b4f00 2024-03-25T18:25:09.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.158+0000 7f74737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:09.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.158+0000 7f74737fe640 1 -- 172.21.15.16:0/3509290817 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7480004860 con 0x7f748c0b4f00 2024-03-25T18:25:09.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.158+0000 7f74737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:09.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.158+0000 7f74737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:09.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.158+0000 7f74737fe640 10 monclient: dump: 2024-03-25T18:25:09.160 INFO:teuthology.orchestra.run.smithi016.stderr:epoch 1 2024-03-25T18:25:09.160 INFO:teuthology.orchestra.run.smithi016.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:09.160 INFO:teuthology.orchestra.run.smithi016.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:09.160 INFO:teuthology.orchestra.run.smithi016.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:09.160 INFO:teuthology.orchestra.run.smithi016.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:09.160 INFO:teuthology.orchestra.run.smithi016.stderr:election_strategy: 3 2024-03-25T18:25:09.160 INFO:teuthology.orchestra.run.smithi016.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:09.160 INFO:teuthology.orchestra.run.smithi016.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:09.160 INFO:teuthology.orchestra.run.smithi016.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:09.160 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:25:09.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.158+0000 7f74737fe640 1 -- 172.21.15.16:0/3509290817 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7480004e10 con 0x7f748c0b4f00 2024-03-25T18:25:09.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.158+0000 7f74897fa640 4 set_mon_vals no callback set 2024-03-25T18:25:09.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.158+0000 7f74737fe640 1 -- 172.21.15.16:0/3509290817 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 2103392613 0 0) 0x7f7480006630 con 0x7f748c0b4f00 2024-03-25T18:25:09.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.158+0000 7f74737fe640 10 client.4153.objecter ms_dispatch 0x7f748c000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T18:25:09.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.158+0000 7f74737fe640 3 client.4153.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T18:25:09.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.158+0000 7f74737fe640 3 client.4153.objecter handle_osd_map decoding full epoch 9 2024-03-25T18:25:09.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.158+0000 7f74737fe640 20 client.4153.objecter dump_active .. 0 homeless 2024-03-25T18:25:09.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.158+0000 7f74737fe640 1 -- 172.21.15.16:0/3509290817 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7480006a30 con 0x7f748c0b4f00 2024-03-25T18:25:09.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.158+0000 7f74737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:09.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.158+0000 7f74737fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:09.231 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-03-25T18:25:09.230+0000 7f49c1ff1040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:25:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.231+0000 7f749144a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c", "id": 7}] 2024-03-25T18:25:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.231+0000 7f749144a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c", "id": 7}] 2024-03-25T18:25:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.231+0000 7f749144a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.231+0000 7f749144a640 1 -- 172.21.15.16:0/3509290817 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd new", "uuid": "98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c", "id": 7} v 0) v1 -- 0x7f748c04f730 con 0x7f748c0b4f00 2024-03-25T18:25:09.241 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-25T18:25:09.240+0000 7f0c86a58040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:25:09.284 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-03-25T18:25:09.284+0000 7f49c1ff1040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:25:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.348+0000 7f74737fe640 1 -- 172.21.15.16:0/3509290817 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 576559068 0 1371665406) 0x7f7480006bd0 con 0x7f748c0b4f00 2024-03-25T18:25:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.348+0000 7f74737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c", "id": 7}] 2024-03-25T18:25:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.348+0000 7f74737fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:09.350 INFO:teuthology.orchestra.run.smithi016.stdout:7 2024-03-25T18:25:09.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.349+0000 7f749144a640 10 librados: watch_flush enter 2024-03-25T18:25:09.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.349+0000 7f749144a640 10 librados: watch_flush exit 2024-03-25T18:25:09.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.349+0000 7f749144a640 20 client.4153.objecter shutdown clearing up homeless session... 2024-03-25T18:25:09.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.349+0000 7f749144a640 10 client.4153.objecter successfully canceled tick 2024-03-25T18:25:09.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.349+0000 7f749144a640 10 monclient: shutdown 2024-03-25T18:25:09.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.349+0000 7f749144a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:09.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.349+0000 7f749144a640 1 -- 172.21.15.16:0/3509290817 >> v1:172.21.15.16:6789/0 conn(0x7f748c0b4f00 legacy=0x7f748c144c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:09.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.349+0000 7f749144a640 1 -- 172.21.15.16:0/3509290817 shutdown_connections 2024-03-25T18:25:09.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.349+0000 7f749144a640 1 -- 172.21.15.16:0/3509290817 >> 172.21.15.16:0/3509290817 conn(0x7f748c066b00 msgr2=0x7f748c1402c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:09.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.349+0000 7f749144a640 1 -- 172.21.15.16:0/3509290817 shutdown_connections 2024-03-25T18:25:09.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.349+0000 7f749144a640 1 -- 172.21.15.16:0/3509290817 wait complete. 2024-03-25T18:25:09.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-25T18:25:09.349+0000 7f749144a640 1 librados: shutdown 2024-03-25T18:25:09.361 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:25:09.361 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T18:25:09.361 DEBUG:teuthology.orchestra.run.smithi003:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T18:25:09.363 INFO:tasks.ceph.osd.0:Started 2024-03-25T18:25:09.364 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:25:09.364 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T18:25:09.364 DEBUG:teuthology.orchestra.run.smithi003:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T18:25:09.366 INFO:tasks.ceph.osd.1:Started 2024-03-25T18:25:09.366 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:25:09.366 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T18:25:09.366 DEBUG:teuthology.orchestra.run.smithi003:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T18:25:09.368 INFO:tasks.ceph.osd.2:Started 2024-03-25T18:25:09.368 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:25:09.368 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T18:25:09.368 DEBUG:teuthology.orchestra.run.smithi003:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T18:25:09.371 INFO:tasks.ceph.osd.3:Started 2024-03-25T18:25:09.371 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:25:09.371 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T18:25:09.371 DEBUG:teuthology.orchestra.run.smithi016:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T18:25:09.403 INFO:tasks.ceph.osd.4:Started 2024-03-25T18:25:09.403 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:25:09.403 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T18:25:09.403 DEBUG:teuthology.orchestra.run.smithi016:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T18:25:09.405 INFO:tasks.ceph.osd.5:Started 2024-03-25T18:25:09.405 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:25:09.405 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T18:25:09.406 DEBUG:teuthology.orchestra.run.smithi016:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T18:25:09.408 INFO:tasks.ceph.osd.6:Started 2024-03-25T18:25:09.408 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:25:09.408 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T18:25:09.408 DEBUG:teuthology.orchestra.run.smithi016:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T18:25:09.410 INFO:tasks.ceph.osd.7:Started 2024-03-25T18:25:09.410 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:09.411 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-03-25T18:25:09.410+0000 7f49c1ff1040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:25:09.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.532+0000 7fd4a292b640 10 monclient: get_monmap_and_config 2024-03-25T18:25:09.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.532+0000 7fd4a292b640 10 monclient: build_initial_monmap 2024-03-25T18:25:09.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.532+0000 7fd4a292b640 10 monclient: monmap: 2024-03-25T18:25:09.534 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-03-25T18:25:09.534 INFO:teuthology.orchestra.run.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:09.534 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-03-25T18:25:09.533946+0000 2024-03-25T18:25:09.534 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-03-25T18:25:09.533946+0000 2024-03-25T18:25:09.534 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:09.534 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:09.534 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:09.534 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:25:09.535 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:25:09.535 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-25T18:25:09.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.533+0000 7fd4a292b640 1 Processor -- start 2024-03-25T18:25:09.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.534+0000 7fd4a292b640 1 -- start start 2024-03-25T18:25:09.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.534+0000 7fd4a292b640 10 monclient: init 2024-03-25T18:25:09.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.534+0000 7fd4a292b640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:09.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.534+0000 7fd4a292b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:09.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.534+0000 7fd4a292b640 10 monclient(hunting): picked mon.noname-c con 0x7fd49c0b4b50 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:09.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.535+0000 7fd4a292b640 10 monclient(hunting): picked mon.noname-b con 0x7fd49c13ff20 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:09.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.535+0000 7fd4a292b640 10 monclient(hunting): picked mon.noname-a con 0x7fd49c142dd0 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:09.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.535+0000 7fd4a292b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd49c0ab440 con 0x7fd49c142dd0 2024-03-25T18:25:09.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.535+0000 7fd4a292b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd49c0ab6b0 con 0x7fd49c0b4b50 2024-03-25T18:25:09.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.535+0000 7fd4a292b640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd49c0a9f10 con 0x7fd49c13ff20 2024-03-25T18:25:09.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.535+0000 7fd4a292b640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:09.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.535+0000 7fd4a292b640 10 monclient(hunting): authenticate will time out at 7363.795494s 2024-03-25T18:25:09.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.540+0000 7fd49bfff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd49c0b4b50 0x7fd49c13f810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:38708/0 (socket says 172.21.15.3:38708) 2024-03-25T18:25:09.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.540+0000 7fd49bfff640 1 -- 172.21.15.3:0/433884746 learned_addr learned my addr 172.21.15.3:0/433884746 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:25:09.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.558+0000 7fd49affd640 1 -- 172.21.15.3:0/433884746 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873525696 0 0) 0x7fd49c0ab440 con 0x7fd49c142dd0 2024-03-25T18:25:09.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.558+0000 7fd49affd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:09.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.558+0000 7fd49affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:09.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.558+0000 7fd49affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:09.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.558+0000 7fd49affd640 10 monclient(hunting): my global_id is 4164 2024-03-25T18:25:09.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.558+0000 7fd49affd640 1 -- 172.21.15.3:0/433884746 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd484003520 con 0x7fd49c142dd0 2024-03-25T18:25:09.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.562+0000 7fd49affd640 1 -- 172.21.15.3:0/433884746 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3437438494 0 0) 0x7fd484003520 con 0x7fd49c142dd0 2024-03-25T18:25:09.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.562+0000 7fd49affd640 1 -- 172.21.15.3:0/433884746 >> v1:172.21.15.3:6790/0 conn(0x7fd49c13ff20 legacy=0x7fd49c1426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:25:09.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.562+0000 7fd49affd640 1 -- 172.21.15.3:0/433884746 >> v1:172.21.15.16:6789/0 conn(0x7fd49c0b4b50 legacy=0x7fd49c13f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:25:09.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.562+0000 7fd49affd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:09.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.562+0000 7fd49affd640 1 monclient: found mon.noname-a 2024-03-25T18:25:09.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.562+0000 7fd49affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:09.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.562+0000 7fd49affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:25:09.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.562+0000 7fd49affd640 1 -- 172.21.15.3:0/433884746 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd49c0aa0a0 con 0x7fd49c142dd0 2024-03-25T18:25:09.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.562+0000 7fd49affd640 10 monclient: _finish_auth 0 2024-03-25T18:25:09.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.562+0000 7fd49affd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:09.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.562+0000 7fd49affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:09.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.562+0000 7fd49affd640 1 -- 172.21.15.3:0/433884746 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd494001a10 con 0x7fd49c142dd0 2024-03-25T18:25:09.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.562+0000 7fd49affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:09.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.562+0000 7fd49affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:09.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.562+0000 7fd49affd640 10 monclient: dump: 2024-03-25T18:25:09.564 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-03-25T18:25:09.564 INFO:teuthology.orchestra.run.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:09.564 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:09.564 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:09.564 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:09.564 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:09.564 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:09.565 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:09.565 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:09.565 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-25T18:25:09.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.562+0000 7fd4a292b640 5 monclient: authenticate success, global_id 4164 2024-03-25T18:25:09.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.562+0000 7fd4a292b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:09.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.564+0000 7fd49affd640 1 -- 172.21.15.3:0/433884746 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd494001b70 con 0x7fd49c142dd0 2024-03-25T18:25:09.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.564+0000 7fd49affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:09.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.564+0000 7fd49affd640 1 -- 172.21.15.3:0/433884746 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd494001fc0 con 0x7fd49c142dd0 2024-03-25T18:25:09.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.564+0000 7fd49affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:09.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.564+0000 7fd49affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:09.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.564+0000 7fd49affd640 10 monclient: dump: 2024-03-25T18:25:09.566 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-03-25T18:25:09.566 INFO:teuthology.orchestra.run.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:09.566 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:09.566 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:09.566 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:09.566 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:09.567 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:09.567 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:09.567 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:09.567 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-25T18:25:09.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.564+0000 7fd4a292b640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:09.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.564+0000 7fd4a292b640 4 set_mon_vals no callback set 2024-03-25T18:25:09.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.565+0000 7fd4a292b640 10 monclient: shutdown 2024-03-25T18:25:09.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.565+0000 7fd4a292b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:09.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.565+0000 7fd4a292b640 1 -- 172.21.15.3:0/433884746 >> v1:172.21.15.3:6789/0 conn(0x7fd49c142dd0 legacy=0x7fd49c14df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:09.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.565+0000 7fd4a292b640 1 -- 172.21.15.3:0/433884746 shutdown_connections 2024-03-25T18:25:09.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.565+0000 7fd4a292b640 1 -- 172.21.15.3:0/433884746 >> 172.21.15.3:0/433884746 conn(0x7fd49c066920 msgr2=0x7fd49c066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:09.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.565+0000 7fd4a292b640 1 -- 172.21.15.3:0/433884746 shutdown_connections 2024-03-25T18:25:09.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.565+0000 7fd4a292b640 1 -- 172.21.15.3:0/433884746 wait complete. 2024-03-25T18:25:09.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.566+0000 7fd4a292b640 10 monclient: build_initial_monmap 2024-03-25T18:25:09.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.566+0000 7fd4a292b640 10 monclient: monmap: 2024-03-25T18:25:09.570 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-03-25T18:25:09.570 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:09.570 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-03-25T18:25:09.570 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-03-25T18:25:09.570 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:09.570 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-03-25T18:25:09.570 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:09.570 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:25:09.570 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:25:09.570 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-25T18:25:09.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.566+0000 7fd4a292b640 1 librados: starting msgr at 2024-03-25T18:25:09.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.566+0000 7fd4a292b640 1 librados: starting objecter 2024-03-25T18:25:09.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.566+0000 7fd4a292b640 1 Processor -- start 2024-03-25T18:25:09.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.567+0000 7fd4a292b640 1 -- start start 2024-03-25T18:25:09.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.567+0000 7fd4a292b640 1 librados: setting wanted keys 2024-03-25T18:25:09.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.567+0000 7fd4a292b640 1 librados: calling monclient init 2024-03-25T18:25:09.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.567+0000 7fd4a292b640 10 monclient: init 2024-03-25T18:25:09.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.567+0000 7fd4a292b640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:09.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.567+0000 7fd4a292b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:09.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.567+0000 7fd4a292b640 10 monclient(hunting): picked mon.noname-c con 0x7fd49c0b4b50 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:09.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.567+0000 7fd4a292b640 10 monclient(hunting): picked mon.noname-a con 0x7fd49c13ff20 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:09.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.567+0000 7fd4a292b640 10 monclient(hunting): picked mon.noname-b con 0x7fd49c142dd0 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:09.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.567+0000 7fd4a292b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd49c141350 con 0x7fd49c13ff20 2024-03-25T18:25:09.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.567+0000 7fd4a292b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd49c138030 con 0x7fd49c142dd0 2024-03-25T18:25:09.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.568+0000 7fd4a292b640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd49c138310 con 0x7fd49c0b4b50 2024-03-25T18:25:09.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.568+0000 7fd4a292b640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:09.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.568+0000 7fd4a292b640 10 monclient(hunting): authenticate will time out at 7363.828387s 2024-03-25T18:25:09.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.568+0000 7fd49b7fe640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd49c13ff20 0x7fd49c138a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50586/0 (socket says 172.21.15.3:50586) 2024-03-25T18:25:09.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.568+0000 7fd49b7fe640 1 -- 172.21.15.3:0/1114143609 learned_addr learned my addr 172.21.15.3:0/1114143609 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:25:09.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.569+0000 7fd498ff9640 10 client.?.objecter ms_handle_connect 0x7fd49c13ff20 2024-03-25T18:25:09.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.569+0000 7fd498ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:09.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.571+0000 7fd498ff9640 1 -- 172.21.15.3:0/1114143609 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738215453 0 0) 0x7fd49c141350 con 0x7fd49c13ff20 2024-03-25T18:25:09.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.571+0000 7fd498ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:09.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.571+0000 7fd498ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:09.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.571+0000 7fd498ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:09.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.571+0000 7fd498ff9640 10 monclient(hunting): my global_id is 4167 2024-03-25T18:25:09.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.571+0000 7fd498ff9640 1 -- 172.21.15.3:0/1114143609 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd488004eb0 con 0x7fd49c13ff20 2024-03-25T18:25:09.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.576+0000 7fd498ff9640 1 -- 172.21.15.3:0/1114143609 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 806887223 0 0) 0x7fd488004eb0 con 0x7fd49c13ff20 2024-03-25T18:25:09.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.576+0000 7fd498ff9640 1 -- 172.21.15.3:0/1114143609 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd49c141350 con 0x7fd49c13ff20 2024-03-25T18:25:09.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.576+0000 7fd498ff9640 1 -- 172.21.15.3:0/1114143609 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd490001960 con 0x7fd49c13ff20 2024-03-25T18:25:09.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.576+0000 7fd498ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:09.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.576+0000 7fd498ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:09.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.576+0000 7fd498ff9640 10 monclient(hunting): dump: 2024-03-25T18:25:09.578 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-03-25T18:25:09.578 INFO:teuthology.orchestra.run.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:09.578 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:09.578 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:09.579 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:09.579 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:09.579 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:09.579 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:09.579 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:09.579 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-25T18:25:09.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.577+0000 7fd498ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:09.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.580+0000 7fd498ff9640 1 -- 172.21.15.3:0/1114143609 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3249772852 0 0) 0x7fd49c141350 con 0x7fd49c13ff20 2024-03-25T18:25:09.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.580+0000 7fd498ff9640 1 -- 172.21.15.3:0/1114143609 >> v1:172.21.15.3:6790/0 conn(0x7fd49c0b4b50 legacy=0x7fd49c134850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:09.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.580+0000 7fd498ff9640 1 -- 172.21.15.3:0/1114143609 >> v1:172.21.15.16:6789/0 conn(0x7fd49c142dd0 legacy=0x7fd49c139120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:09.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.581+0000 7fd498ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:25:09.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.581+0000 7fd498ff9640 1 monclient: found mon.a 2024-03-25T18:25:09.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.581+0000 7fd498ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:09.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.581+0000 7fd498ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:09.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.581+0000 7fd498ff9640 1 -- 172.21.15.3:0/1114143609 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd49c1385f0 con 0x7fd49c13ff20 2024-03-25T18:25:09.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.581+0000 7fd498ff9640 10 monclient: _finish_auth 0 2024-03-25T18:25:09.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.581+0000 7fd498ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:25:09.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.581+0000 7fd498ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:09.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.581+0000 7fd4a292b640 5 monclient: authenticate success, global_id 4167 2024-03-25T18:25:09.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.581+0000 7fd4a292b640 10 monclient: _renew_subs 2024-03-25T18:25:09.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.581+0000 7fd4a292b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:09.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.581+0000 7fd4a292b640 1 -- 172.21.15.3:0/1114143609 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd49c134f60 con 0x7fd49c13ff20 2024-03-25T18:25:09.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.581+0000 7fd4a292b640 10 client.4167.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:09.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.581+0000 7fd4a292b640 10 monclient: _renew_subs 2024-03-25T18:25:09.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.581+0000 7fd4a292b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:09.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.581+0000 7fd4a292b640 1 -- 172.21.15.3:0/1114143609 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd49c137390 con 0x7fd49c13ff20 2024-03-25T18:25:09.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.581+0000 7fd4a292b640 1 librados: init done 2024-03-25T18:25:09.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.582+0000 7fd4a292b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:09.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.582+0000 7fd4a292b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:09.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.582+0000 7fd4a292b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:09.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.582+0000 7fd4a292b640 1 -- 172.21.15.3:0/1114143609 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd49c0b79d0 con 0x7fd49c13ff20 2024-03-25T18:25:09.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.714+0000 7fd498ff9640 1 -- 172.21.15.3:0/1114143609 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd490001ac0 con 0x7fd49c13ff20 2024-03-25T18:25:09.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.714+0000 7fd498ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:09.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.714+0000 7fd49a7fc640 4 set_mon_vals no callback set 2024-03-25T18:25:09.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.715+0000 7fd498ff9640 1 -- 172.21.15.3:0/1114143609 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd490003a40 con 0x7fd49c13ff20 2024-03-25T18:25:09.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.715+0000 7fd498ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:09.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.715+0000 7fd498ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:09.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.715+0000 7fd498ff9640 10 monclient: dump: 2024-03-25T18:25:09.717 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-03-25T18:25:09.717 INFO:teuthology.orchestra.run.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:09.717 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:09.717 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:09.717 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:09.717 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:09.717 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:09.717 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:09.717 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:09.717 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-25T18:25:09.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.715+0000 7fd498ff9640 1 -- 172.21.15.3:0/1114143609 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 853387467 0 0) 0x7fd490003c60 con 0x7fd49c13ff20 2024-03-25T18:25:09.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.718+0000 7fd498ff9640 1 -- 172.21.15.3:0/1114143609 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7fd490015820 con 0x7fd49c13ff20 2024-03-25T18:25:09.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.718+0000 7fd498ff9640 10 client.4167.objecter ms_dispatch 0x7fd49c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:09.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.718+0000 7fd498ff9640 3 client.4167.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:09.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.718+0000 7fd498ff9640 3 client.4167.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:09.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.718+0000 7fd498ff9640 20 client.4167.objecter dump_active .. 0 homeless 2024-03-25T18:25:09.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.718+0000 7fd498ff9640 1 -- 172.21.15.3:0/1114143609 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd490014ef0 con 0x7fd49c13ff20 2024-03-25T18:25:09.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.718+0000 7fd498ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:09.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.718+0000 7fd498ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:09.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.797+0000 7fd4a292b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:09.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.797+0000 7fd4a292b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:09.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.797+0000 7fd4a292b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:09.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.797+0000 7fd4a292b640 1 -- 172.21.15.3:0/1114143609 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd49c04f730 con 0x7fd49c13ff20 2024-03-25T18:25:09.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.869+0000 7fd498ff9640 1 -- 172.21.15.3:0/1114143609 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7fd49001d800 con 0x7fd49c13ff20 2024-03-25T18:25:09.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.869+0000 7fd498ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:09.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.869+0000 7fd498ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:09.871 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-25T18:25:09.871 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:09.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.870+0000 7fd4a292b640 10 librados: watch_flush enter 2024-03-25T18:25:09.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.870+0000 7fd4a292b640 10 librados: watch_flush exit 2024-03-25T18:25:09.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.870+0000 7fd4a292b640 20 client.4167.objecter shutdown clearing up homeless session... 2024-03-25T18:25:09.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.870+0000 7fd4a292b640 10 client.4167.objecter successfully canceled tick 2024-03-25T18:25:09.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.870+0000 7fd4a292b640 10 monclient: shutdown 2024-03-25T18:25:09.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.870+0000 7fd4a292b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:09.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.870+0000 7fd4a292b640 1 -- 172.21.15.3:0/1114143609 >> v1:172.21.15.3:6789/0 conn(0x7fd49c13ff20 legacy=0x7fd49c138a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:09.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.870+0000 7fd4a292b640 1 -- 172.21.15.3:0/1114143609 shutdown_connections 2024-03-25T18:25:09.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.870+0000 7fd4a292b640 1 -- 172.21.15.3:0/1114143609 >> 172.21.15.3:0/1114143609 conn(0x7fd49c066920 msgr2=0x7fd49c0b2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:09.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.870+0000 7fd4a292b640 1 -- 172.21.15.3:0/1114143609 shutdown_connections 2024-03-25T18:25:09.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.870+0000 7fd4a292b640 1 -- 172.21.15.3:0/1114143609 wait complete. 2024-03-25T18:25:09.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-25T18:25:09.870+0000 7fd4a292b640 1 librados: shutdown 2024-03-25T18:25:09.884 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T18:25:09.885 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T18:25:10.690 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:10.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.834+0000 7f5eea606640 10 monclient: get_monmap_and_config 2024-03-25T18:25:10.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.834+0000 7f5eea606640 10 monclient: build_initial_monmap 2024-03-25T18:25:10.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.834+0000 7f5eea606640 10 monclient: monmap: 2024-03-25T18:25:10.836 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:25:10.836 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:10.836 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:25:10.836000+0000 2024-03-25T18:25:10.836 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:25:10.836000+0000 2024-03-25T18:25:10.836 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:10.837 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:10.837 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:10.837 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:25:10.837 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:25:10.837 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:10.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.836+0000 7f5eea606640 1 Processor -- start 2024-03-25T18:25:10.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.836+0000 7f5eea606640 1 -- start start 2024-03-25T18:25:10.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.836+0000 7f5eea606640 10 monclient: init 2024-03-25T18:25:10.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.837+0000 7f5eea606640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:10.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.837+0000 7f5eea606640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:10.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.837+0000 7f5eea606640 10 monclient(hunting): picked mon.noname-b con 0x7f5ee40b1ed0 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:10.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.837+0000 7f5eea606640 10 monclient(hunting): picked mon.noname-a con 0x7f5ee414a650 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:10.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.837+0000 7f5eea606640 10 monclient(hunting): picked mon.noname-c con 0x7f5ee4144090 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:10.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.837+0000 7f5eea606640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee406eba0 con 0x7f5ee414a650 2024-03-25T18:25:10.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.837+0000 7f5eea606640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee414a140 con 0x7f5ee4144090 2024-03-25T18:25:10.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.837+0000 7f5eea606640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee406ee10 con 0x7f5ee40b1ed0 2024-03-25T18:25:10.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.837+0000 7f5eea606640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:10.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.837+0000 7f5eea606640 10 monclient(hunting): authenticate will time out at 7365.097825s 2024-03-25T18:25:10.840 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.839+0000 7f5ee8ba3640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5ee4144090 0x7f5ee4152000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:38812/0 (socket says 172.21.15.3:38812) 2024-03-25T18:25:10.840 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.839+0000 7f5ee8ba3640 1 -- 172.21.15.3:0/3611050997 learned_addr learned my addr 172.21.15.3:0/3611050997 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:25:10.913 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-25T18:25:10.912+0000 7f0c27ec2640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T18:25:10.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.913+0000 7f5ee2ffd640 1 -- 172.21.15.3:0/3611050997 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3888587035 0 0) 0x7f5ee406eba0 con 0x7f5ee414a650 2024-03-25T18:25:10.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.913+0000 7f5ee2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:10.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.913+0000 7f5ee2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:10.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.913+0000 7f5ee2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:10.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.913+0000 7f5ee2ffd640 10 monclient(hunting): my global_id is 4188 2024-03-25T18:25:10.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.913+0000 7f5ee2ffd640 1 -- 172.21.15.3:0/3611050997 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ecc003520 con 0x7f5ee414a650 2024-03-25T18:25:10.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.920+0000 7f5ee2ffd640 1 -- 172.21.15.3:0/3611050997 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2339720192 0 0) 0x7f5ecc003520 con 0x7f5ee414a650 2024-03-25T18:25:10.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.920+0000 7f5ee2ffd640 1 -- 172.21.15.3:0/3611050997 >> v1:172.21.15.3:6790/0 conn(0x7f5ee40b1ed0 legacy=0x7f5ee40b2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:25:10.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.920+0000 7f5ee2ffd640 1 -- 172.21.15.3:0/3611050997 >> v1:172.21.15.16:6789/0 conn(0x7f5ee4144090 legacy=0x7f5ee4152000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:10.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.920+0000 7f5ee2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:10.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.920+0000 7f5ee2ffd640 1 monclient: found mon.noname-a 2024-03-25T18:25:10.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.920+0000 7f5ee2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:10.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.920+0000 7f5ee2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:25:10.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.920+0000 7f5ee2ffd640 1 -- 172.21.15.3:0/3611050997 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee406f0f0 con 0x7f5ee414a650 2024-03-25T18:25:10.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.920+0000 7f5ee2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:25:10.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.920+0000 7f5ee2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:10.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.920+0000 7f5ee2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:10.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.920+0000 7f5eea606640 5 monclient: authenticate success, global_id 4188 2024-03-25T18:25:10.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.920+0000 7f5eea606640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:10.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.920+0000 7f5ee2ffd640 1 -- 172.21.15.3:0/3611050997 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f5ed8001690 con 0x7f5ee414a650 2024-03-25T18:25:10.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.920+0000 7f5ee2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:10.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.920+0000 7f5ee2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:10.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.920+0000 7f5ee2ffd640 10 monclient: dump: 2024-03-25T18:25:10.922 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:10.922 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:10.922 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:10.922 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:10.923 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:10.923 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:10.923 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:10.923 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:10.923 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:10.923 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:10.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.920+0000 7f5eea606640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:10.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.922+0000 7f5ee2ffd640 1 -- 172.21.15.3:0/3611050997 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ed8001960 con 0x7f5ee414a650 2024-03-25T18:25:10.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.922+0000 7f5ee2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:10.924 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.922+0000 7f5eea606640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:10.924 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.922+0000 7f5eea606640 4 set_mon_vals no callback set 2024-03-25T18:25:10.924 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.923+0000 7f5eea606640 10 monclient: shutdown 2024-03-25T18:25:10.925 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.923+0000 7f5eea606640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:10.925 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.923+0000 7f5eea606640 1 -- 172.21.15.3:0/3611050997 >> v1:172.21.15.3:6789/0 conn(0x7f5ee414a650 legacy=0x7f5ee4143980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:10.925 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.923+0000 7f5eea606640 1 -- 172.21.15.3:0/3611050997 shutdown_connections 2024-03-25T18:25:10.925 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.923+0000 7f5eea606640 1 -- 172.21.15.3:0/3611050997 >> 172.21.15.3:0/3611050997 conn(0x7f5ee40617a0 msgr2=0x7f5ee4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:10.925 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.924+0000 7f5eea606640 1 -- 172.21.15.3:0/3611050997 shutdown_connections 2024-03-25T18:25:10.925 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.924+0000 7f5eea606640 1 -- 172.21.15.3:0/3611050997 wait complete. 2024-03-25T18:25:10.926 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.925+0000 7f5eea606640 10 monclient: build_initial_monmap 2024-03-25T18:25:10.926 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.925+0000 7f5eea606640 10 monclient: monmap: 2024-03-25T18:25:10.927 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:25:10.927 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:10.927 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:25:10.927 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:25:10.927 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:10.927 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:25:10.927 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:10.927 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:25:10.927 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:25:10.927 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:10.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.925+0000 7f5eea606640 1 librados: starting msgr at 2024-03-25T18:25:10.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.925+0000 7f5eea606640 1 librados: starting objecter 2024-03-25T18:25:10.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.925+0000 7f5eea606640 1 Processor -- start 2024-03-25T18:25:10.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.925+0000 7f5eea606640 1 -- start start 2024-03-25T18:25:10.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.925+0000 7f5eea606640 1 librados: setting wanted keys 2024-03-25T18:25:10.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.925+0000 7f5eea606640 1 librados: calling monclient init 2024-03-25T18:25:10.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.925+0000 7f5eea606640 10 monclient: init 2024-03-25T18:25:10.929 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.926+0000 7f5eea606640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:10.929 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.926+0000 7f5eea606640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:25:10.929 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.926+0000 7f5eea606640 10 monclient(hunting): picked mon.noname-b con 0x7f5ee40b1ed0 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:10.929 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.926+0000 7f5eea606640 10 monclient(hunting): picked mon.noname-a con 0x7f5ee4144090 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:10.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.927+0000 7f5eea606640 10 monclient(hunting): picked mon.noname-c con 0x7f5ee414a650 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:10.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.927+0000 7f5eea606640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee41471f0 con 0x7f5ee4144090 2024-03-25T18:25:10.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.927+0000 7f5eea606640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee406f9a0 con 0x7f5ee40b1ed0 2024-03-25T18:25:10.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.927+0000 7f5eea606640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee413c810 con 0x7f5ee414a650 2024-03-25T18:25:10.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.927+0000 7f5eea606640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:10.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.927+0000 7f5eea606640 10 monclient(hunting): authenticate will time out at 7365.187532s 2024-03-25T18:25:10.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.927+0000 7f5ee37fe640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f5ee4144090 0x7f5ee413c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50688/0 (socket says 172.21.15.3:50688) 2024-03-25T18:25:10.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.927+0000 7f5ee37fe640 1 -- 172.21.15.3:0/61116696 learned_addr learned my addr 172.21.15.3:0/61116696 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:25:10.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.927+0000 7f5ee0ff9640 10 client.?.objecter ms_handle_connect 0x7f5ee4144090 2024-03-25T18:25:10.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.927+0000 7f5ee0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:10.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.927+0000 7f5ee0ff9640 10 client.?.objecter ms_handle_connect 0x7f5ee40b1ed0 2024-03-25T18:25:10.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.927+0000 7f5ee0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:10.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.931+0000 7f5ee0ff9640 1 -- 172.21.15.3:0/61116696 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2609839943 0 0) 0x7f5ee41471f0 con 0x7f5ee4144090 2024-03-25T18:25:10.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.931+0000 7f5ee0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:10.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.931+0000 7f5ee0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:10.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.931+0000 7f5ee0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:10.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.931+0000 7f5ee0ff9640 10 monclient(hunting): my global_id is 4191 2024-03-25T18:25:10.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.931+0000 7f5ee0ff9640 1 -- 172.21.15.3:0/61116696 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ed0004eb0 con 0x7f5ee4144090 2024-03-25T18:25:10.941 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.939+0000 7f5ee0ff9640 1 -- 172.21.15.3:0/61116696 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4163907259 0 0) 0x7f5ed0004eb0 con 0x7f5ee4144090 2024-03-25T18:25:10.941 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.939+0000 7f5ee0ff9640 1 -- 172.21.15.3:0/61116696 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ee41471f0 con 0x7f5ee4144090 2024-03-25T18:25:10.941 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.939+0000 7f5ee0ff9640 1 -- 172.21.15.3:0/61116696 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f5ed8001710 con 0x7f5ee4144090 2024-03-25T18:25:10.941 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.939+0000 7f5ee0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:10.941 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.939+0000 7f5ee0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:10.941 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.939+0000 7f5ee0ff9640 10 monclient(hunting): dump: 2024-03-25T18:25:10.941 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:10.941 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:10.941 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:10.941 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:10.941 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:10.941 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:10.942 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:10.942 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:10.942 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:10.942 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:10.942 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.939+0000 7f5ee0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:10.943 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.941+0000 7f5ee0ff9640 10 client.?.objecter ms_handle_connect 0x7f5ee414a650 2024-03-25T18:25:10.943 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.941+0000 7f5ee0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:10.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.949+0000 7f5ee0ff9640 1 -- 172.21.15.3:0/61116696 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2698804903 0 0) 0x7f5ee41471f0 con 0x7f5ee4144090 2024-03-25T18:25:10.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.949+0000 7f5ee0ff9640 1 -- 172.21.15.3:0/61116696 >> v1:172.21.15.3:6790/0 conn(0x7f5ee414a650 legacy=0x7f5ee41dbda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:10.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.949+0000 7f5ee0ff9640 1 -- 172.21.15.3:0/61116696 >> v1:172.21.15.16:6789/0 conn(0x7f5ee40b1ed0 legacy=0x7f5ee4144e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:10.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.949+0000 7f5ee0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:25:10.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.949+0000 7f5ee0ff9640 1 monclient: found mon.a 2024-03-25T18:25:10.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.949+0000 7f5ee0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:10.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.949+0000 7f5ee0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:10.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.949+0000 7f5ee0ff9640 1 -- 172.21.15.3:0/61116696 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee413c9a0 con 0x7f5ee4144090 2024-03-25T18:25:10.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.949+0000 7f5ee0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:25:10.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.949+0000 7f5ee0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:25:10.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.949+0000 7f5ee0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:10.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.949+0000 7f5eea606640 5 monclient: authenticate success, global_id 4191 2024-03-25T18:25:10.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.949+0000 7f5eea606640 10 monclient: _renew_subs 2024-03-25T18:25:10.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.949+0000 7f5eea606640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:10.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.949+0000 7f5eea606640 1 -- 172.21.15.3:0/61116696 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ee41dc4b0 con 0x7f5ee4144090 2024-03-25T18:25:10.953 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.949+0000 7f5eea606640 10 client.4191.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:10.953 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.949+0000 7f5eea606640 10 monclient: _renew_subs 2024-03-25T18:25:10.953 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.949+0000 7f5eea606640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:10.953 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.949+0000 7f5eea606640 1 -- 172.21.15.3:0/61116696 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ee4145db0 con 0x7f5ee4144090 2024-03-25T18:25:10.953 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.950+0000 7f5eea606640 1 librados: init done 2024-03-25T18:25:10.953 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.951+0000 7f5eea606640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:10.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.951+0000 7f5eea606640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:10.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.951+0000 7f5eea606640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:10.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:10.951+0000 7f5eea606640 1 -- 172.21.15.3:0/61116696 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ee4146100 con 0x7f5ee4144090 2024-03-25T18:25:11.005 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-25T18:25:11.004+0000 7f0c27ec2640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T18:25:11.510 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.507+0000 7f5ee0ff9640 1 -- 172.21.15.3:0/61116696 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ed80038a0 con 0x7f5ee4144090 2024-03-25T18:25:11.510 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.507+0000 7f5ee0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:11.510 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.507+0000 7f5ee0ff9640 1 -- 172.21.15.3:0/61116696 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f5ed8003c70 con 0x7f5ee4144090 2024-03-25T18:25:11.510 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.507+0000 7f5ee0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:11.510 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.507+0000 7f5ee0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:11.510 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.507+0000 7f5ee0ff9640 10 monclient: dump: 2024-03-25T18:25:11.510 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:11.510 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:11.510 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:11.511 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:11.511 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:11.511 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:11.511 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:11.511 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:11.511 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:11.511 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:11.511 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.507+0000 7f5ee27fc640 4 set_mon_vals no callback set 2024-03-25T18:25:11.511 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.508+0000 7f5ee0ff9640 1 -- 172.21.15.3:0/61116696 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 2887383621 0 0) 0x7f5ed8003e90 con 0x7f5ee4144090 2024-03-25T18:25:11.511 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.508+0000 7f5ee3fff640 10 monclient: get_auth_request con 0x7f5ed006db70 auth_method 0 2024-03-25T18:25:11.511 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.509+0000 7f5ee0ff9640 10 client.4191.objecter ms_handle_connect 0x7f5ed006db70 2024-03-25T18:25:11.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.546+0000 7f5ee0ff9640 1 -- 172.21.15.3:0/61116696 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f5ed80548c0 con 0x7f5ee4144090 2024-03-25T18:25:11.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.546+0000 7f5ee0ff9640 10 client.4191.objecter ms_dispatch 0x7f5ee4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:11.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.546+0000 7f5ee0ff9640 3 client.4191.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:11.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.546+0000 7f5ee0ff9640 3 client.4191.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:11.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.546+0000 7f5ee0ff9640 20 client.4191.objecter dump_active .. 0 homeless 2024-03-25T18:25:11.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.546+0000 7f5ee0ff9640 1 -- 172.21.15.3:0/61116696 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5ed80503d0 con 0x7f5ee4144090 2024-03-25T18:25:11.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.546+0000 7f5ee0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:11.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.546+0000 7f5ee0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:11.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.888+0000 7f5eea606640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:11.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.888+0000 7f5eea606640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:11.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.888+0000 7f5eea606640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:11.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.888+0000 7f5eea606640 1 -- 172.21.15.3:0/61116696 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ee414aaa0 con 0x7f5ee4144090 2024-03-25T18:25:11.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.914+0000 7f5ee0ff9640 1 -- 172.21.15.3:0/61116696 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f5ee414aaa0 con 0x7f5ee4144090 2024-03-25T18:25:11.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.914+0000 7f5ee0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:11.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.914+0000 7f5ee0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:11.916 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:25:11.917 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:11.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.917+0000 7f5eea606640 10 librados: watch_flush enter 2024-03-25T18:25:11.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.917+0000 7f5eea606640 10 librados: watch_flush exit 2024-03-25T18:25:11.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.917+0000 7f5eea606640 20 client.4191.objecter shutdown clearing up homeless session... 2024-03-25T18:25:11.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.917+0000 7f5eea606640 10 client.4191.objecter successfully canceled tick 2024-03-25T18:25:11.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.918+0000 7f5eea606640 1 -- 172.21.15.3:0/61116696 >> v1:172.21.15.3:6800/4195140966 conn(0x7f5ed006db70 legacy=0x7f5ed0070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:11.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.918+0000 7f5eea606640 10 monclient: shutdown 2024-03-25T18:25:11.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.918+0000 7f5eea606640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:11.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.918+0000 7f5eea606640 1 -- 172.21.15.3:0/61116696 >> v1:172.21.15.3:6789/0 conn(0x7f5ee4144090 legacy=0x7f5ee413c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:11.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.918+0000 7f5eea606640 1 -- 172.21.15.3:0/61116696 shutdown_connections 2024-03-25T18:25:11.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.918+0000 7f5eea606640 1 -- 172.21.15.3:0/61116696 >> 172.21.15.3:0/61116696 conn(0x7f5ee40617a0 msgr2=0x7f5ee413d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:11.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.918+0000 7f5eea606640 1 -- 172.21.15.3:0/61116696 shutdown_connections 2024-03-25T18:25:11.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.918+0000 7f5eea606640 1 -- 172.21.15.3:0/61116696 wait complete. 2024-03-25T18:25:11.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:11.918+0000 7f5eea606640 1 librados: shutdown 2024-03-25T18:25:11.936 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:16.417 INFO:tasks.ceph.osd.5.smithi016.stderr:==00:00:00:06.947 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:16.456 INFO:tasks.ceph.osd.4.smithi016.stderr:==00:00:00:06.963 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:16.481 INFO:tasks.ceph.osd.5.smithi016.stderr:==00:00:00:07.012 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:16.486 INFO:tasks.ceph.osd.7.smithi016.stderr:==00:00:00:07.011 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:16.519 INFO:tasks.ceph.osd.4.smithi016.stderr:==00:00:00:07.026 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:16.547 INFO:tasks.ceph.osd.7.smithi016.stderr:==00:00:00:07.072 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:16.678 INFO:tasks.ceph.osd.6.smithi016.stderr:==00:00:00:07.207 35188== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:16.703 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:07.268 35484== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:16.736 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:07.295 35485== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:16.738 INFO:tasks.ceph.osd.6.smithi016.stderr:==00:00:00:07.267 35188== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:16.772 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:07.338 35484== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:16.799 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:07.358 35485== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:17.030 INFO:tasks.ceph.osd.5.smithi016.stderr:2024-03-25T18:25:17.021+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:25:17.073 INFO:tasks.ceph.osd.4.smithi016.stderr:2024-03-25T18:25:17.066+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:25:17.112 INFO:tasks.ceph.osd.7.smithi016.stderr:2024-03-25T18:25:17.105+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:25:17.319 INFO:tasks.ceph.osd.1.smithi003.stderr:2024-03-25T18:25:17.309+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:25:17.320 INFO:tasks.ceph.osd.6.smithi016.stderr:2024-03-25T18:25:17.313+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:25:17.352 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:07.919 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:17.356 INFO:tasks.ceph.osd.2.smithi003.stderr:2024-03-25T18:25:17.349+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:25:17.400 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:07.942 35490== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:17.418 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:07.985 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:17.462 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:08.003 35490== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:17.495 INFO:tasks.ceph.osd.5.smithi016.stderr:==00:00:00:08.025 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:17.535 INFO:tasks.ceph.osd.4.smithi016.stderr:==00:00:00:08.043 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:17.654 INFO:tasks.ceph.osd.7.smithi016.stderr:==00:00:00:08.179 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:17.808 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:08.373 35484== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:17.826 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:08.385 35485== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:17.840 INFO:tasks.ceph.osd.6.smithi016.stderr:==00:00:00:08.370 35188== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:17.871 INFO:tasks.ceph.osd.5.smithi016.stderr:==00:00:00:08.402 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:17.912 INFO:tasks.ceph.osd.4.smithi016.stderr:==00:00:00:08.419 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:17.988 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-03-25T18:25:17.981+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:25:18.011 INFO:tasks.ceph.osd.3.smithi003.stderr:2024-03-25T18:25:18.004+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:25:18.026 INFO:tasks.ceph.osd.7.smithi016.stderr:==00:00:00:08.551 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:18.193 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:08.758 35484== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:18.197 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:08.756 35485== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:18.210 INFO:tasks.ceph.osd.6.smithi016.stderr:==00:00:00:08.740 35188== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:18.227 INFO:tasks.ceph.osd.4.smithi016.stderr:==00:00:00:08.735 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:18.229 INFO:tasks.ceph.osd.5.smithi016.stderr:==00:00:00:08.760 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:18.351 INFO:tasks.ceph.osd.7.smithi016.stderr:==00:00:00:08.876 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:18.473 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:09.040 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:18.475 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:09.016 35490== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:18.493 INFO:tasks.ceph.osd.4.smithi016.stderr:==00:00:00:09.000 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:18.517 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:09.076 35485== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:18.521 INFO:tasks.ceph.osd.5.smithi016.stderr:==00:00:00:09.051 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:18.531 INFO:tasks.ceph.osd.6.smithi016.stderr:==00:00:00:09.061 35188== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:18.559 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:09.124 35484== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:18.617 INFO:tasks.ceph.osd.7.smithi016.stderr:==00:00:00:09.142 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:18.741 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:18.760 INFO:tasks.ceph.osd.4.smithi016.stderr:==00:00:00:09.267 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:18.789 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:09.348 35485== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:18.800 INFO:tasks.ceph.osd.6.smithi016.stderr:==00:00:00:09.329 35188== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:18.817 INFO:tasks.ceph.osd.5.smithi016.stderr:==00:00:00:09.347 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:18.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.831+0000 7efd3d033640 10 monclient: get_monmap_and_config 2024-03-25T18:25:18.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.831+0000 7efd3d033640 10 monclient: build_initial_monmap 2024-03-25T18:25:18.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.831+0000 7efd3d033640 10 monclient: monmap: 2024-03-25T18:25:18.833 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:25:18.833 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:18.833 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:25:18.832561+0000 2024-03-25T18:25:18.833 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:25:18.832561+0000 2024-03-25T18:25:18.833 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:18.833 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:18.833 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:18.834 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:25:18.834 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:25:18.834 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:18.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.832+0000 7efd3d033640 1 Processor -- start 2024-03-25T18:25:18.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.832+0000 7efd3d033640 1 -- start start 2024-03-25T18:25:18.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.832+0000 7efd3d033640 10 monclient: init 2024-03-25T18:25:18.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.833+0000 7efd3d033640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:18.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.833+0000 7efd3d033640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:25:18.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.833+0000 7efd3d033640 10 monclient(hunting): picked mon.noname-c con 0x7efd380b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:18.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.833+0000 7efd3d033640 10 monclient(hunting): picked mon.noname-a con 0x7efd380b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:18.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.833+0000 7efd3d033640 10 monclient(hunting): picked mon.noname-b con 0x7efd3813c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:18.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.833+0000 7efd3d033640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd380ab0e0 con 0x7efd380b4670 2024-03-25T18:25:18.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.833+0000 7efd3d033640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd380ab3c0 con 0x7efd380b4ff0 2024-03-25T18:25:18.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.833+0000 7efd3d033640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd380a9f30 con 0x7efd3813c480 2024-03-25T18:25:18.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.833+0000 7efd3d033640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:18.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.833+0000 7efd3d033640 10 monclient(hunting): authenticate will time out at 7373.093475s 2024-03-25T18:25:18.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.833+0000 7efd36d9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7efd380b4ff0 0x7efd3813bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:41262/0 (socket says 172.21.15.3:41262) 2024-03-25T18:25:18.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.833+0000 7efd36d9d640 1 -- 172.21.15.3:0/934958356 learned_addr learned my addr 172.21.15.3:0/934958356 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:25:18.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.835+0000 7efd35d9b640 1 -- 172.21.15.3:0/934958356 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454475428 0 0) 0x7efd380ab0e0 con 0x7efd380b4670 2024-03-25T18:25:18.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.835+0000 7efd35d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:25:18.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.835+0000 7efd35d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:18.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.835+0000 7efd35d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:18.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.835+0000 7efd35d9b640 10 monclient(hunting): my global_id is 4218 2024-03-25T18:25:18.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.835+0000 7efd35d9b640 1 -- 172.21.15.3:0/934958356 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd20003520 con 0x7efd380b4670 2024-03-25T18:25:18.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.835+0000 7efd35d9b640 1 -- 172.21.15.3:0/934958356 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1820768602 0 0) 0x7efd380ab3c0 con 0x7efd380b4ff0 2024-03-25T18:25:18.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.835+0000 7efd35d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:25:18.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.835+0000 7efd35d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:18.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.835+0000 7efd35d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:18.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.835+0000 7efd35d9b640 10 monclient(hunting): my global_id is 4183 2024-03-25T18:25:18.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.835+0000 7efd35d9b640 1 -- 172.21.15.3:0/934958356 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd380ab0e0 con 0x7efd380b4ff0 2024-03-25T18:25:18.840 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.837+0000 7efd35d9b640 1 -- 172.21.15.3:0/934958356 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1241357066 0 0) 0x7efd380a9f30 con 0x7efd3813c480 2024-03-25T18:25:18.840 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.837+0000 7efd35d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:25:18.840 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.837+0000 7efd35d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:18.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.837+0000 7efd35d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:18.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.837+0000 7efd35d9b640 10 monclient(hunting): my global_id is 4190 2024-03-25T18:25:18.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.837+0000 7efd35d9b640 1 -- 172.21.15.3:0/934958356 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd380ab3c0 con 0x7efd3813c480 2024-03-25T18:25:18.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.838+0000 7efd35d9b640 1 -- 172.21.15.3:0/934958356 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 930642955 0 0) 0x7efd380ab0e0 con 0x7efd380b4ff0 2024-03-25T18:25:18.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.838+0000 7efd35d9b640 1 -- 172.21.15.3:0/934958356 >> v1:172.21.15.3:6790/0 conn(0x7efd3813c480 legacy=0x7efd3814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:18.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.838+0000 7efd35d9b640 1 -- 172.21.15.3:0/934958356 >> v1:172.21.15.3:6789/0 conn(0x7efd380b4670 legacy=0x7efd381425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:18.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.838+0000 7efd35d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:25:18.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.838+0000 7efd35d9b640 1 monclient: found mon.noname-c 2024-03-25T18:25:18.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.838+0000 7efd35d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:18.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.838+0000 7efd35d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:25:18.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.838+0000 7efd35d9b640 1 -- 172.21.15.3:0/934958356 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd380a9b70 con 0x7efd380b4ff0 2024-03-25T18:25:18.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.838+0000 7efd35d9b640 10 monclient: _finish_auth 0 2024-03-25T18:25:18.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.838+0000 7efd35d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:25:18.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.838+0000 7efd35d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:18.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.838+0000 7efd3d033640 5 monclient: authenticate success, global_id 4183 2024-03-25T18:25:18.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.838+0000 7efd3d033640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:18.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.839+0000 7efd35d9b640 1 -- 172.21.15.3:0/934958356 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7efd28001630 con 0x7efd380b4ff0 2024-03-25T18:25:18.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.839+0000 7efd35d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:18.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.839+0000 7efd35d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:18.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.839+0000 7efd35d9b640 10 monclient: dump: 2024-03-25T18:25:18.842 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:18.842 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:18.842 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:18.842 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:18.842 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:18.842 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:18.842 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:18.843 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:18.843 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:18.843 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:18.843 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.839+0000 7efd3d033640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:18.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.840+0000 7efd35d9b640 1 -- 172.21.15.3:0/934958356 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd28001880 con 0x7efd380b4ff0 2024-03-25T18:25:18.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.840+0000 7efd35d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:18.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.840+0000 7efd3d033640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:18.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.840+0000 7efd3d033640 4 set_mon_vals no callback set 2024-03-25T18:25:18.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.840+0000 7efd35d9b640 1 -- 172.21.15.3:0/934958356 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7efd28001c10 con 0x7efd380b4ff0 2024-03-25T18:25:18.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.840+0000 7efd35d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:18.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.840+0000 7efd35d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:18.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.840+0000 7efd35d9b640 10 monclient: dump: 2024-03-25T18:25:18.847 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:18.847 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:18.847 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:18.847 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:18.847 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:18.847 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:18.847 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:18.847 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:18.847 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:18.848 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:18.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.841+0000 7efd3d033640 10 monclient: shutdown 2024-03-25T18:25:18.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.841+0000 7efd3d033640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:18.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.841+0000 7efd3d033640 1 -- 172.21.15.3:0/934958356 >> v1:172.21.15.16:6789/0 conn(0x7efd380b4ff0 legacy=0x7efd3813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:18.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.841+0000 7efd3d033640 1 -- 172.21.15.3:0/934958356 shutdown_connections 2024-03-25T18:25:18.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.841+0000 7efd3d033640 1 -- 172.21.15.3:0/934958356 >> 172.21.15.3:0/934958356 conn(0x7efd380617a0 msgr2=0x7efd38061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:18.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.841+0000 7efd3d033640 1 -- 172.21.15.3:0/934958356 shutdown_connections 2024-03-25T18:25:18.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.841+0000 7efd3d033640 1 -- 172.21.15.3:0/934958356 wait complete. 2024-03-25T18:25:18.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.842+0000 7efd3d033640 10 monclient: build_initial_monmap 2024-03-25T18:25:18.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.842+0000 7efd3d033640 10 monclient: monmap: 2024-03-25T18:25:18.848 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:25:18.848 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:18.848 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:25:18.849 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:25:18.849 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:18.849 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:25:18.849 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:18.849 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:25:18.849 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:25:18.849 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:18.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.842+0000 7efd3d033640 1 librados: starting msgr at 2024-03-25T18:25:18.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.842+0000 7efd3d033640 1 librados: starting objecter 2024-03-25T18:25:18.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.842+0000 7efd3d033640 1 Processor -- start 2024-03-25T18:25:18.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.842+0000 7efd3d033640 1 -- start start 2024-03-25T18:25:18.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.842+0000 7efd3d033640 1 librados: setting wanted keys 2024-03-25T18:25:18.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.842+0000 7efd3d033640 1 librados: calling monclient init 2024-03-25T18:25:18.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.842+0000 7efd3d033640 10 monclient: init 2024-03-25T18:25:18.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.843+0000 7efd3d033640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:18.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.843+0000 7efd3d033640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:18.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.843+0000 7efd3d033640 10 monclient(hunting): picked mon.noname-c con 0x7efd380b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:18.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.843+0000 7efd3d033640 10 monclient(hunting): picked mon.noname-a con 0x7efd380b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:18.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.843+0000 7efd3d033640 10 monclient(hunting): picked mon.noname-b con 0x7efd3813c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:18.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.843+0000 7efd3d033640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd38138860 con 0x7efd380b4ff0 2024-03-25T18:25:18.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.843+0000 7efd3d033640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd381dac10 con 0x7efd3813c480 2024-03-25T18:25:18.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.843+0000 7efd3d033640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd381db840 con 0x7efd380b4670 2024-03-25T18:25:18.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.843+0000 7efd3d033640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:18.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.843+0000 7efd3d033640 10 monclient(hunting): authenticate will time out at 7373.103718s 2024-03-25T18:25:18.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.843+0000 7efd3759e640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7efd3813c480 0x7efd381e2e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:41264/0 (socket says 172.21.15.3:41264) 2024-03-25T18:25:18.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.843+0000 7efd3759e640 1 -- 172.21.15.3:0/51261900 learned_addr learned my addr 172.21.15.3:0/51261900 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:25:18.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.844+0000 7efd1f7fe640 10 client.?.objecter ms_handle_connect 0x7efd3813c480 2024-03-25T18:25:18.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.844+0000 7efd1f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:18.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.844+0000 7efd1f7fe640 10 client.?.objecter ms_handle_connect 0x7efd380b4670 2024-03-25T18:25:18.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.844+0000 7efd1f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:18.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.844+0000 7efd1f7fe640 10 client.?.objecter ms_handle_connect 0x7efd380b4ff0 2024-03-25T18:25:18.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.844+0000 7efd1f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:18.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.845+0000 7efd1f7fe640 1 -- 172.21.15.3:0/51261900 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 650977499 0 0) 0x7efd381dac10 con 0x7efd3813c480 2024-03-25T18:25:18.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.845+0000 7efd1f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:18.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.845+0000 7efd1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:18.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.845+0000 7efd1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:18.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.845+0000 7efd1f7fe640 10 monclient(hunting): my global_id is 4186 2024-03-25T18:25:18.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.845+0000 7efd1f7fe640 1 -- 172.21.15.3:0/51261900 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd24004eb0 con 0x7efd3813c480 2024-03-25T18:25:18.852 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.846+0000 7efd1f7fe640 1 -- 172.21.15.3:0/51261900 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3938065596 0 0) 0x7efd38138860 con 0x7efd380b4ff0 2024-03-25T18:25:18.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.846+0000 7efd1f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:18.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.846+0000 7efd1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:18.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.846+0000 7efd1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:18.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.846+0000 7efd1f7fe640 10 monclient(hunting): my global_id is 4221 2024-03-25T18:25:18.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.846+0000 7efd1f7fe640 1 -- 172.21.15.3:0/51261900 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd381dac10 con 0x7efd380b4ff0 2024-03-25T18:25:18.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.847+0000 7efd1f7fe640 1 -- 172.21.15.3:0/51261900 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1993868259 0 0) 0x7efd381db840 con 0x7efd380b4670 2024-03-25T18:25:18.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.847+0000 7efd1f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:18.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.847+0000 7efd1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:18.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.847+0000 7efd1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:18.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.847+0000 7efd1f7fe640 10 monclient(hunting): my global_id is 4193 2024-03-25T18:25:18.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.847+0000 7efd1f7fe640 1 -- 172.21.15.3:0/51261900 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd38138860 con 0x7efd380b4670 2024-03-25T18:25:18.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.850+0000 7efd1f7fe640 1 -- 172.21.15.3:0/51261900 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1929008719 0 0) 0x7efd24004eb0 con 0x7efd3813c480 2024-03-25T18:25:18.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.850+0000 7efd1f7fe640 1 -- 172.21.15.3:0/51261900 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd381db840 con 0x7efd3813c480 2024-03-25T18:25:18.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.850+0000 7efd1f7fe640 1 -- 172.21.15.3:0/51261900 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7efd30001c70 con 0x7efd3813c480 2024-03-25T18:25:18.854 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.850+0000 7efd1f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:18.854 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.850+0000 7efd1f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:18.854 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.850+0000 7efd1f7fe640 10 monclient(hunting): dump: 2024-03-25T18:25:18.854 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:18.854 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:18.854 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:18.854 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:18.854 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:18.854 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:18.854 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:18.854 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:18.855 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:18.855 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:18.855 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.850+0000 7efd1f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:18.855 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.850+0000 7efd1f7fe640 1 -- 172.21.15.3:0/51261900 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2955644951 0 0) 0x7efd381dac10 con 0x7efd380b4ff0 2024-03-25T18:25:18.855 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.850+0000 7efd1f7fe640 1 -- 172.21.15.3:0/51261900 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd24004eb0 con 0x7efd380b4ff0 2024-03-25T18:25:18.855 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.850+0000 7efd1f7fe640 1 -- 172.21.15.3:0/51261900 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7efd2c001950 con 0x7efd380b4ff0 2024-03-25T18:25:18.855 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.850+0000 7efd1f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:18.855 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.850+0000 7efd1f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:18.855 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.850+0000 7efd1f7fe640 10 monclient(hunting): dump: 2024-03-25T18:25:18.855 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:18.855 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:18.855 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:18.855 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:18.855 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:18.856 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:18.856 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:18.856 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:18.856 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:18.856 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:18.858 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.853+0000 7efd1f7fe640 1 -- 172.21.15.3:0/51261900 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1853841879 0 0) 0x7efd381db840 con 0x7efd3813c480 2024-03-25T18:25:18.858 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.853+0000 7efd1f7fe640 1 -- 172.21.15.3:0/51261900 >> v1:172.21.15.3:6790/0 conn(0x7efd380b4670 legacy=0x7efd381db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:18.858 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.853+0000 7efd1f7fe640 1 -- 172.21.15.3:0/51261900 >> v1:172.21.15.3:6789/0 conn(0x7efd380b4ff0 legacy=0x7efd381d46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:18.858 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.853+0000 7efd1f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:18.858 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.853+0000 7efd1f7fe640 1 monclient: found mon.b 2024-03-25T18:25:18.858 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.853+0000 7efd1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:18.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.853+0000 7efd1f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:18.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.853+0000 7efd1f7fe640 1 -- 172.21.15.3:0/51261900 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd381db9d0 con 0x7efd3813c480 2024-03-25T18:25:18.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.853+0000 7efd1f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:18.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.853+0000 7efd1f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:18.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.853+0000 7efd1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:18.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.853+0000 7efd3d033640 5 monclient: authenticate success, global_id 4186 2024-03-25T18:25:18.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.853+0000 7efd3d033640 10 monclient: _renew_subs 2024-03-25T18:25:18.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.853+0000 7efd3d033640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:18.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.853+0000 7efd3d033640 1 -- 172.21.15.3:0/51261900 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd38137840 con 0x7efd3813c480 2024-03-25T18:25:18.859 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:09.420 35484== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:18.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.853+0000 7efd3d033640 10 client.4186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:18.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.853+0000 7efd3d033640 10 monclient: _renew_subs 2024-03-25T18:25:18.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.853+0000 7efd3d033640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:18.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.853+0000 7efd3d033640 1 -- 172.21.15.3:0/51261900 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd381457f0 con 0x7efd3813c480 2024-03-25T18:25:18.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.853+0000 7efd3d033640 1 librados: init done 2024-03-25T18:25:18.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.854+0000 7efd3d033640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:18.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.854+0000 7efd3d033640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:18.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.855+0000 7efd3d033640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:18.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.855+0000 7efd3d033640 1 -- 172.21.15.3:0/51261900 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd3806dbd0 con 0x7efd3813c480 2024-03-25T18:25:18.862 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.855+0000 7efd1f7fe640 1 -- 172.21.15.3:0/51261900 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd30003990 con 0x7efd3813c480 2024-03-25T18:25:18.862 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.855+0000 7efd1f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:18.862 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.855+0000 7efd3559a640 4 set_mon_vals no callback set 2024-03-25T18:25:18.862 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:09.402 35490== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:18.862 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:09.429 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:18.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.869+0000 7efd1f7fe640 1 -- 172.21.15.3:0/51261900 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7efd30003b80 con 0x7efd3813c480 2024-03-25T18:25:18.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.869+0000 7efd1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:18.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.869+0000 7efd1f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:18.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.869+0000 7efd1f7fe640 10 monclient: dump: 2024-03-25T18:25:18.871 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:18.871 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:18.871 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:18.871 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:18.871 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:18.871 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:18.871 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:18.871 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:18.871 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:18.871 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:18.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.870+0000 7efd1f7fe640 1 -- 172.21.15.3:0/51261900 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7efd30003d60 con 0x7efd3813c480 2024-03-25T18:25:18.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.871+0000 7efd36d9d640 10 monclient: get_auth_request con 0x7efd2406df70 auth_method 0 2024-03-25T18:25:18.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.871+0000 7efd1f7fe640 10 client.4186.objecter ms_handle_connect 0x7efd2406df70 2024-03-25T18:25:18.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.873+0000 7efd1f7fe640 1 -- 172.21.15.3:0/51261900 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7efd30054940 con 0x7efd3813c480 2024-03-25T18:25:18.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.873+0000 7efd1f7fe640 10 client.4186.objecter ms_dispatch 0x7efd38000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:18.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.873+0000 7efd1f7fe640 3 client.4186.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:18.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.873+0000 7efd1f7fe640 3 client.4186.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:18.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:18.873+0000 7efd1f7fe640 20 client.4186.objecter dump_active .. 0 homeless 2024-03-25T18:25:18.887 INFO:tasks.ceph.osd.7.smithi016.stderr:==00:00:00:09.412 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:19.027 INFO:tasks.ceph.osd.4.smithi016.stderr:==00:00:00:09.535 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:19.057 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:09.616 35485== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:19.073 INFO:tasks.ceph.osd.6.smithi016.stderr:==00:00:00:09.602 35188== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:19.112 INFO:tasks.ceph.osd.5.smithi016.stderr:==00:00:00:09.642 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:19.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:19.121+0000 7efd1f7fe640 1 -- 172.21.15.3:0/51261900 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efd30050450 con 0x7efd3813c480 2024-03-25T18:25:19.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:19.121+0000 7efd1f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:19.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:19.121+0000 7efd1f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:19.148 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:09.713 35484== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:19.158 INFO:tasks.ceph.osd.7.smithi016.stderr:==00:00:00:09.683 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:19.191 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:09.759 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:19.226 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:09.767 35490== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:19.291 INFO:tasks.ceph.osd.4.smithi016.stderr:==00:00:00:09.799 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:19.328 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:09.887 35485== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:19.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:19.330+0000 7efd3d033640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:19.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:19.330+0000 7efd3d033640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:19.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:19.330+0000 7efd3d033640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:19.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:19.330+0000 7efd3d033640 1 -- 172.21.15.3:0/51261900 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd380688c0 con 0x7efd3813c480 2024-03-25T18:25:19.344 INFO:tasks.ceph.osd.6.smithi016.stderr:==00:00:00:09.874 35188== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:19.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:19.376+0000 7efd1f7fe640 1 -- 172.21.15.3:0/51261900 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7efd30052380 con 0x7efd3813c480 2024-03-25T18:25:19.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:19.376+0000 7efd1f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:19.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:19.376+0000 7efd1f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:19.377 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:25:19.377 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:19.379 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:19.378+0000 7efd3d033640 10 librados: watch_flush enter 2024-03-25T18:25:19.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:19.378+0000 7efd3d033640 10 librados: watch_flush exit 2024-03-25T18:25:19.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:19.378+0000 7efd3d033640 20 client.4186.objecter shutdown clearing up homeless session... 2024-03-25T18:25:19.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:19.378+0000 7efd3d033640 10 client.4186.objecter successfully canceled tick 2024-03-25T18:25:19.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:19.378+0000 7efd3d033640 1 -- 172.21.15.3:0/51261900 >> v1:172.21.15.3:6800/4195140966 conn(0x7efd2406df70 legacy=0x7efd24070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:19.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:19.378+0000 7efd3d033640 10 monclient: shutdown 2024-03-25T18:25:19.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:19.378+0000 7efd3d033640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:19.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:19.378+0000 7efd3d033640 1 -- 172.21.15.3:0/51261900 >> v1:172.21.15.16:6789/0 conn(0x7efd3813c480 legacy=0x7efd381e2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:19.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:19.378+0000 7efd3d033640 1 -- 172.21.15.3:0/51261900 shutdown_connections 2024-03-25T18:25:19.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:19.378+0000 7efd3d033640 1 -- 172.21.15.3:0/51261900 >> 172.21.15.3:0/51261900 conn(0x7efd380617a0 msgr2=0x7efd38141b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:19.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:19.378+0000 7efd3d033640 1 -- 172.21.15.3:0/51261900 shutdown_connections 2024-03-25T18:25:19.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:19.378+0000 7efd3d033640 1 -- 172.21.15.3:0/51261900 wait complete. 2024-03-25T18:25:19.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:19.378+0000 7efd3d033640 1 librados: shutdown 2024-03-25T18:25:19.388 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:19.404 INFO:tasks.ceph.osd.5.smithi016.stderr:==00:00:00:09.934 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:19.429 INFO:tasks.ceph.osd.7.smithi016.stderr:==00:00:00:09.954 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:19.462 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:10.030 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:19.463 INFO:tasks.ceph.osd.5.smithi016.stderr:==00:00:00:09.993 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:19.469 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:10.034 35484== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:19.519 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:10.060 35490== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:19.559 INFO:tasks.ceph.osd.4.smithi016.stderr:==00:00:00:10.067 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:19.593 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:10.152 35485== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:19.616 INFO:tasks.ceph.osd.6.smithi016.stderr:==00:00:00:10.145 35188== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:19.693 INFO:tasks.ceph.osd.7.smithi016.stderr:==00:00:00:10.218 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:19.729 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:10.297 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:19.755 INFO:tasks.ceph.osd.5.smithi016.stderr:==00:00:00:10.285 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:19.768 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:10.333 35484== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:19.811 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:10.352 35490== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:19.831 INFO:tasks.ceph.osd.4.smithi016.stderr:==00:00:00:10.338 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:19.857 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:10.416 35485== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:19.880 INFO:tasks.ceph.osd.6.smithi016.stderr:==00:00:00:10.410 35188== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:19.961 INFO:tasks.ceph.osd.7.smithi016.stderr:==00:00:00:10.486 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:20.001 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:10.569 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:20.057 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:10.622 35484== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:20.100 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:10.641 35490== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:20.122 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:10.681 35485== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:20.149 INFO:tasks.ceph.osd.6.smithi016.stderr:==00:00:00:10.678 35188== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:20.266 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:10.834 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:20.279 INFO:tasks.ceph.osd.5.smithi016.stderr:==00:00:00:10.810 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:20.312 INFO:tasks.ceph.osd.4.smithi016.stderr:==00:00:00:10.820 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:20.335 INFO:tasks.ceph.osd.5.smithi016.stderr:==00:00:00:10.866 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:20.352 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:10.917 35484== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:20.368 INFO:tasks.ceph.osd.4.smithi016.stderr:==00:00:00:10.876 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:20.402 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:10.944 35490== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:20.435 INFO:tasks.ceph.osd.7.smithi016.stderr:==00:00:00:10.960 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:20.491 INFO:tasks.ceph.osd.7.smithi016.stderr:==00:00:00:11.016 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:20.533 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:11.101 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:20.621 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:11.179 35485== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:20.677 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:11.236 35485== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:20.690 INFO:tasks.ceph.osd.6.smithi016.stderr:==00:00:00:11.219 35188== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:20.712 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:11.253 35490== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:20.758 INFO:tasks.ceph.osd.6.smithi016.stderr:==00:00:00:11.287 35188== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:20.799 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:11.367 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:20.877 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:11.442 35484== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:20.934 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:11.499 35484== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:21.003 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:11.544 35490== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:21.278 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:11.845 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:21.362 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:11.929 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:21.596 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:12.137 35490== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:21.652 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:12.193 35490== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:25.789 INFO:tasks.ceph.osd.5.smithi016.stderr:==00:00:00:16.320 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:26.041 INFO:tasks.ceph.osd.4.smithi016.stderr:==00:00:00:16.548 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:26.182 INFO:tasks.ceph.osd.7.smithi016.stderr:==00:00:00:16.707 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:26.196 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:26.231 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:16.790 35485== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:26.297 INFO:tasks.ceph.osd.6.smithi016.stderr:==00:00:00:16.826 35188== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:26.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.315+0000 7f5e78c93640 10 monclient: get_monmap_and_config 2024-03-25T18:25:26.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.315+0000 7f5e78c93640 10 monclient: build_initial_monmap 2024-03-25T18:25:26.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.315+0000 7f5e78c93640 10 monclient: monmap: 2024-03-25T18:25:26.316 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:25:26.316 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:26.317 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:25:26.316602+0000 2024-03-25T18:25:26.317 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:25:26.316602+0000 2024-03-25T18:25:26.317 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:26.317 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:26.317 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:26.317 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:25:26.317 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:25:26.317 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:26.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.316+0000 7f5e78c93640 1 Processor -- start 2024-03-25T18:25:26.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.316+0000 7f5e78c93640 1 -- start start 2024-03-25T18:25:26.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.316+0000 7f5e78c93640 10 monclient: init 2024-03-25T18:25:26.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.316+0000 7f5e78c93640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:26.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.316+0000 7f5e78c93640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:26.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.316+0000 7f5e78c93640 10 monclient(hunting): picked mon.noname-b con 0x7f5e740b4e70 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:26.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.316+0000 7f5e78c93640 10 monclient(hunting): picked mon.noname-a con 0x7f5e740b44b0 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:26.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.316+0000 7f5e78c93640 10 monclient(hunting): picked mon.noname-c con 0x7f5e740b1b00 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:26.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.316+0000 7f5e78c93640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e740a96f0 con 0x7f5e740b44b0 2024-03-25T18:25:26.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.316+0000 7f5e78c93640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e740ab0e0 con 0x7f5e740b1b00 2024-03-25T18:25:26.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.316+0000 7f5e78c93640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e74145f70 con 0x7f5e740b4e70 2024-03-25T18:25:26.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.316+0000 7f5e78c93640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:26.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.316+0000 7f5e78c93640 10 monclient(hunting): authenticate will time out at 7380.577258s 2024-03-25T18:25:26.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.317+0000 7f5e7259c640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f5e740b4e70 0x7f5e7413f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:43804/0 (socket says 172.21.15.3:43804) 2024-03-25T18:25:26.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.317+0000 7f5e7259c640 1 -- 172.21.15.3:0/2006345783 learned_addr learned my addr 172.21.15.3:0/2006345783 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:25:26.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.320+0000 7f5e7159a640 1 -- 172.21.15.3:0/2006345783 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362847488 0 0) 0x7f5e74145f70 con 0x7f5e740b4e70 2024-03-25T18:25:26.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.320+0000 7f5e7159a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:26.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.320+0000 7f5e7159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:26.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.320+0000 7f5e7159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:26.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.320+0000 7f5e7159a640 10 monclient(hunting): my global_id is 4196 2024-03-25T18:25:26.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.320+0000 7f5e7159a640 1 -- 172.21.15.3:0/2006345783 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e5c003520 con 0x7f5e740b4e70 2024-03-25T18:25:26.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.321+0000 7f5e7159a640 1 -- 172.21.15.3:0/2006345783 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1223684158 0 0) 0x7f5e740ab0e0 con 0x7f5e740b1b00 2024-03-25T18:25:26.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.321+0000 7f5e7159a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:26.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.321+0000 7f5e7159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:26.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.321+0000 7f5e7159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:26.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.321+0000 7f5e7159a640 10 monclient(hunting): my global_id is 4189 2024-03-25T18:25:26.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.321+0000 7f5e7159a640 1 -- 172.21.15.3:0/2006345783 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e5c003840 con 0x7f5e740b1b00 2024-03-25T18:25:26.324 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.322+0000 7f5e7159a640 1 -- 172.21.15.3:0/2006345783 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3553205442 0 0) 0x7f5e740a96f0 con 0x7f5e740b44b0 2024-03-25T18:25:26.324 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.322+0000 7f5e7159a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:26.324 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.322+0000 7f5e7159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:26.324 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.322+0000 7f5e7159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:26.324 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.322+0000 7f5e7159a640 10 monclient(hunting): my global_id is 4224 2024-03-25T18:25:26.324 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.322+0000 7f5e7159a640 1 -- 172.21.15.3:0/2006345783 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e740ab0e0 con 0x7f5e740b44b0 2024-03-25T18:25:26.326 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.323+0000 7f5e7159a640 1 -- 172.21.15.3:0/2006345783 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 716830841 0 0) 0x7f5e5c003520 con 0x7f5e740b4e70 2024-03-25T18:25:26.326 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.323+0000 7f5e7159a640 1 -- 172.21.15.3:0/2006345783 >> v1:172.21.15.16:6789/0 conn(0x7f5e740b1b00 legacy=0x7f5e7414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:26.326 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.323+0000 7f5e7159a640 1 -- 172.21.15.3:0/2006345783 >> v1:172.21.15.3:6789/0 conn(0x7f5e740b44b0 legacy=0x7f5e74145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:26.326 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.323+0000 7f5e7159a640 10 monclient: _finish_hunting 0 2024-03-25T18:25:26.326 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.323+0000 7f5e7159a640 1 monclient: found mon.noname-b 2024-03-25T18:25:26.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.323+0000 7f5e7159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:26.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.323+0000 7f5e7159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-25T18:25:26.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.323+0000 7f5e7159a640 1 -- 172.21.15.3:0/2006345783 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e740a9b80 con 0x7f5e740b4e70 2024-03-25T18:25:26.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.323+0000 7f5e7159a640 10 monclient: _finish_auth 0 2024-03-25T18:25:26.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.323+0000 7f5e7159a640 10 monclient: _check_auth_tickets 2024-03-25T18:25:26.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.323+0000 7f5e7159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:26.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.324+0000 7f5e78c93640 5 monclient: authenticate success, global_id 4196 2024-03-25T18:25:26.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.324+0000 7f5e78c93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:26.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.324+0000 7f5e7159a640 1 -- 172.21.15.3:0/2006345783 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f5e64001670 con 0x7f5e740b4e70 2024-03-25T18:25:26.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.324+0000 7f5e7159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:26.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.324+0000 7f5e7159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:26.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.324+0000 7f5e7159a640 10 monclient: dump: 2024-03-25T18:25:26.327 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:26.327 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:26.328 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:26.328 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:26.328 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:26.328 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:26.328 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:26.328 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:26.328 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:26.328 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:26.328 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.324+0000 7f5e78c93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:26.329 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.326+0000 7f5e7159a640 1 -- 172.21.15.3:0/2006345783 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e64001850 con 0x7f5e740b4e70 2024-03-25T18:25:26.330 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.326+0000 7f5e7159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:26.330 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.326+0000 7f5e7159a640 1 -- 172.21.15.3:0/2006345783 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f5e64001c60 con 0x7f5e740b4e70 2024-03-25T18:25:26.330 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.326+0000 7f5e7159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:26.330 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.326+0000 7f5e7159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:26.330 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.326+0000 7f5e7159a640 10 monclient: dump: 2024-03-25T18:25:26.330 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:26.330 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:26.330 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:26.330 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:26.330 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:26.330 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:26.330 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:26.331 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:26.331 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:26.331 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:26.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.326+0000 7f5e78c93640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:26.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.326+0000 7f5e78c93640 4 set_mon_vals no callback set 2024-03-25T18:25:26.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.327+0000 7f5e78c93640 10 monclient: shutdown 2024-03-25T18:25:26.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.327+0000 7f5e78c93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:26.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.327+0000 7f5e78c93640 1 -- 172.21.15.3:0/2006345783 >> v1:172.21.15.3:6790/0 conn(0x7f5e740b4e70 legacy=0x7f5e7413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:26.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.327+0000 7f5e78c93640 1 -- 172.21.15.3:0/2006345783 shutdown_connections 2024-03-25T18:25:26.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.327+0000 7f5e78c93640 1 -- 172.21.15.3:0/2006345783 >> 172.21.15.3:0/2006345783 conn(0x7f5e740617a0 msgr2=0x7f5e74061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:26.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.327+0000 7f5e78c93640 1 -- 172.21.15.3:0/2006345783 shutdown_connections 2024-03-25T18:25:26.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.327+0000 7f5e78c93640 1 -- 172.21.15.3:0/2006345783 wait complete. 2024-03-25T18:25:26.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.328+0000 7f5e78c93640 10 monclient: build_initial_monmap 2024-03-25T18:25:26.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.329+0000 7f5e78c93640 10 monclient: monmap: 2024-03-25T18:25:26.332 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:25:26.332 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:26.332 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:25:26.332 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:25:26.332 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:26.332 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:25:26.332 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:26.332 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:25:26.332 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:25:26.332 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:26.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.329+0000 7f5e78c93640 1 librados: starting msgr at 2024-03-25T18:25:26.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.329+0000 7f5e78c93640 1 librados: starting objecter 2024-03-25T18:25:26.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.329+0000 7f5e78c93640 1 Processor -- start 2024-03-25T18:25:26.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.329+0000 7f5e78c93640 1 -- start start 2024-03-25T18:25:26.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.329+0000 7f5e78c93640 1 librados: setting wanted keys 2024-03-25T18:25:26.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.329+0000 7f5e78c93640 1 librados: calling monclient init 2024-03-25T18:25:26.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.329+0000 7f5e78c93640 10 monclient: init 2024-03-25T18:25:26.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.330+0000 7f5e78c93640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:26.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.330+0000 7f5e78c93640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:26.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.330+0000 7f5e78c93640 10 monclient(hunting): picked mon.noname-a con 0x7f5e740b1b00 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:26.336 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.330+0000 7f5e78c93640 10 monclient(hunting): picked mon.noname-c con 0x7f5e740b44b0 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:26.336 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.330+0000 7f5e78c93640 10 monclient(hunting): picked mon.noname-b con 0x7f5e740b4e70 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:26.336 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.330+0000 7f5e78c93640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e7413c580 con 0x7f5e740b1b00 2024-03-25T18:25:26.336 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.330+0000 7f5e78c93640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e741dbb50 con 0x7f5e740b4e70 2024-03-25T18:25:26.336 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.330+0000 7f5e78c93640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e74130230 con 0x7f5e740b44b0 2024-03-25T18:25:26.336 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.330+0000 7f5e78c93640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:26.336 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.330+0000 7f5e78c93640 10 monclient(hunting): authenticate will time out at 7380.591048s 2024-03-25T18:25:26.336 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.331+0000 7f5e71d9b640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f5e740b44b0 0x7f5e7412e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:43810/0 (socket says 172.21.15.3:43810) 2024-03-25T18:25:26.336 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.331+0000 7f5e71d9b640 1 -- 172.21.15.3:0/2764134820 learned_addr learned my addr 172.21.15.3:0/2764134820 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:25:26.336 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.331+0000 7f5e62ffd640 10 client.?.objecter ms_handle_connect 0x7f5e740b44b0 2024-03-25T18:25:26.336 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.331+0000 7f5e62ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:26.336 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.331+0000 7f5e62ffd640 10 client.?.objecter ms_handle_connect 0x7f5e740b4e70 2024-03-25T18:25:26.336 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.331+0000 7f5e62ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:26.336 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.331+0000 7f5e62ffd640 10 client.?.objecter ms_handle_connect 0x7f5e740b1b00 2024-03-25T18:25:26.337 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.331+0000 7f5e62ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:26.337 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.333+0000 7f5e62ffd640 1 -- 172.21.15.3:0/2764134820 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1406708232 0 0) 0x7f5e74130230 con 0x7f5e740b44b0 2024-03-25T18:25:26.337 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.333+0000 7f5e62ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:26.337 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.333+0000 7f5e62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:26.337 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.333+0000 7f5e62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:26.337 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.333+0000 7f5e62ffd640 10 monclient(hunting): my global_id is 4199 2024-03-25T18:25:26.337 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.333+0000 7f5e62ffd640 1 -- 172.21.15.3:0/2764134820 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e54004eb0 con 0x7f5e740b44b0 2024-03-25T18:25:26.337 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.333+0000 7f5e62ffd640 1 -- 172.21.15.3:0/2764134820 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 644453212 0 0) 0x7f5e741dbb50 con 0x7f5e740b4e70 2024-03-25T18:25:26.337 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.333+0000 7f5e62ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:26.337 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.333+0000 7f5e62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:26.337 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.333+0000 7f5e62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:26.337 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.333+0000 7f5e62ffd640 10 monclient(hunting): my global_id is 4192 2024-03-25T18:25:26.337 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.333+0000 7f5e62ffd640 1 -- 172.21.15.3:0/2764134820 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e74130230 con 0x7f5e740b4e70 2024-03-25T18:25:26.338 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.335+0000 7f5e62ffd640 1 -- 172.21.15.3:0/2764134820 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 254185918 0 0) 0x7f5e7413c580 con 0x7f5e740b1b00 2024-03-25T18:25:26.338 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.335+0000 7f5e62ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:26.338 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.335+0000 7f5e62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:26.338 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.335+0000 7f5e62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:26.338 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.335+0000 7f5e62ffd640 10 monclient(hunting): my global_id is 4227 2024-03-25T18:25:26.338 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.335+0000 7f5e62ffd640 1 -- 172.21.15.3:0/2764134820 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e741dbb50 con 0x7f5e740b1b00 2024-03-25T18:25:26.339 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.338+0000 7f5e62ffd640 1 -- 172.21.15.3:0/2764134820 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 770398031 0 0) 0x7f5e54004eb0 con 0x7f5e740b44b0 2024-03-25T18:25:26.339 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.338+0000 7f5e62ffd640 1 -- 172.21.15.3:0/2764134820 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e54006680 con 0x7f5e740b44b0 2024-03-25T18:25:26.340 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.338+0000 7f5e62ffd640 1 -- 172.21.15.3:0/2764134820 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f5e68001a70 con 0x7f5e740b44b0 2024-03-25T18:25:26.340 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.338+0000 7f5e62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:26.340 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.338+0000 7f5e62ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:26.340 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.338+0000 7f5e62ffd640 10 monclient(hunting): dump: 2024-03-25T18:25:26.340 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:26.340 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:26.340 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:26.340 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:26.340 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:26.340 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:26.340 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:26.341 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:26.341 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:26.341 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:26.341 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.338+0000 7f5e62ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:26.341 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.338+0000 7f5e62ffd640 1 -- 172.21.15.3:0/2764134820 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 264139733 0 0) 0x7f5e74130230 con 0x7f5e740b4e70 2024-03-25T18:25:26.341 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.338+0000 7f5e62ffd640 1 -- 172.21.15.3:0/2764134820 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e54004eb0 con 0x7f5e740b4e70 2024-03-25T18:25:26.341 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.338+0000 7f5e62ffd640 1 -- 172.21.15.3:0/2764134820 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f5e6c001cb0 con 0x7f5e740b4e70 2024-03-25T18:25:26.341 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.338+0000 7f5e62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:26.341 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.338+0000 7f5e62ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:26.341 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.338+0000 7f5e62ffd640 10 monclient(hunting): dump: 2024-03-25T18:25:26.341 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:26.341 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:26.341 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:26.341 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:26.341 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:26.341 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:26.342 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:26.342 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:26.342 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:26.342 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:26.343 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.342+0000 7f5e62ffd640 1 -- 172.21.15.3:0/2764134820 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4210140746 0 0) 0x7f5e54004eb0 con 0x7f5e740b4e70 2024-03-25T18:25:26.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.342+0000 7f5e62ffd640 1 -- 172.21.15.3:0/2764134820 >> v1:172.21.15.3:6790/0 conn(0x7f5e740b44b0 legacy=0x7f5e7412e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:26.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.342+0000 7f5e62ffd640 1 -- 172.21.15.3:0/2764134820 >> v1:172.21.15.3:6789/0 conn(0x7f5e740b1b00 legacy=0x7f5e741db440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:26.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.342+0000 7f5e62ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:26.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.342+0000 7f5e62ffd640 1 monclient: found mon.b 2024-03-25T18:25:26.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.342+0000 7f5e62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:26.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.342+0000 7f5e62ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:26.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.342+0000 7f5e62ffd640 1 -- 172.21.15.3:0/2764134820 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e7413c7d0 con 0x7f5e740b4e70 2024-03-25T18:25:26.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.342+0000 7f5e62ffd640 10 monclient: _finish_auth 0 2024-03-25T18:25:26.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.342+0000 7f5e62ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:26.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.342+0000 7f5e62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:26.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.342+0000 7f5e78c93640 5 monclient: authenticate success, global_id 4192 2024-03-25T18:25:26.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.342+0000 7f5e78c93640 10 monclient: _renew_subs 2024-03-25T18:25:26.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.342+0000 7f5e78c93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:26.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.342+0000 7f5e78c93640 1 -- 172.21.15.3:0/2764134820 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e7413ca40 con 0x7f5e740b4e70 2024-03-25T18:25:26.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.342+0000 7f5e78c93640 10 client.4192.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:26.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.342+0000 7f5e78c93640 10 monclient: _renew_subs 2024-03-25T18:25:26.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.342+0000 7f5e78c93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:26.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.342+0000 7f5e78c93640 1 -- 172.21.15.3:0/2764134820 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e7412f2c0 con 0x7f5e740b4e70 2024-03-25T18:25:26.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.342+0000 7f5e78c93640 1 librados: init done 2024-03-25T18:25:26.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.344+0000 7f5e78c93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:26.346 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.344+0000 7f5e78c93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:26.346 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.344+0000 7f5e78c93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:26.346 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.344+0000 7f5e78c93640 1 -- 172.21.15.3:0/2764134820 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e7413cc30 con 0x7f5e740b4e70 2024-03-25T18:25:26.368 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:16.933 35484== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:26.665 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.663+0000 7f5e62ffd640 1 -- 172.21.15.3:0/2764134820 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e6c0020c0 con 0x7f5e740b4e70 2024-03-25T18:25:26.665 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.663+0000 7f5e62ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:26.665 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.663+0000 7f5e62ffd640 1 -- 172.21.15.3:0/2764134820 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f5e6c003b30 con 0x7f5e740b4e70 2024-03-25T18:25:26.665 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.663+0000 7f5e62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:26.665 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.663+0000 7f5e62ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:26.665 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.663+0000 7f5e62ffd640 10 monclient: dump: 2024-03-25T18:25:26.665 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:26.665 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:26.665 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:26.665 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:26.666 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:26.666 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:26.666 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:26.666 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:26.666 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:26.666 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:26.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.663+0000 7f5e70d99640 4 set_mon_vals no callback set 2024-03-25T18:25:26.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.664+0000 7f5e62ffd640 1 -- 172.21.15.3:0/2764134820 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f5e6c019b00 con 0x7f5e740b4e70 2024-03-25T18:25:26.667 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.664+0000 7f5e7259c640 10 monclient: get_auth_request con 0x7f5e5406e100 auth_method 0 2024-03-25T18:25:26.667 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.665+0000 7f5e62ffd640 10 client.4192.objecter ms_handle_connect 0x7f5e5406e100 2024-03-25T18:25:26.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.672+0000 7f5e62ffd640 1 -- 172.21.15.3:0/2764134820 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f5e6c053760 con 0x7f5e740b4e70 2024-03-25T18:25:26.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.672+0000 7f5e62ffd640 10 client.4192.objecter ms_dispatch 0x7f5e74000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:26.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.672+0000 7f5e62ffd640 3 client.4192.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:26.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.672+0000 7f5e62ffd640 3 client.4192.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:26.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.672+0000 7f5e62ffd640 20 client.4192.objecter dump_active .. 0 homeless 2024-03-25T18:25:26.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.672+0000 7f5e62ffd640 1 -- 172.21.15.3:0/2764134820 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5e6c04f0a0 con 0x7f5e740b4e70 2024-03-25T18:25:26.674 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.672+0000 7f5e62ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:26.674 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.672+0000 7f5e62ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:26.693 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:17.261 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:26.858 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.857+0000 7f5e78c93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:26.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.857+0000 7f5e78c93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:26.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.857+0000 7f5e78c93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:26.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.857+0000 7f5e78c93640 1 -- 172.21.15.3:0/2764134820 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e7404f730 con 0x7f5e740b4e70 2024-03-25T18:25:26.883 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.881+0000 7f5e62ffd640 1 -- 172.21.15.3:0/2764134820 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f5e6c0511a0 con 0x7f5e740b4e70 2024-03-25T18:25:26.883 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.881+0000 7f5e62ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:26.883 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.881+0000 7f5e62ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:26.883 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:25:26.883 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:26.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.883+0000 7f5e78c93640 10 librados: watch_flush enter 2024-03-25T18:25:26.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.883+0000 7f5e78c93640 10 librados: watch_flush exit 2024-03-25T18:25:26.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.883+0000 7f5e78c93640 20 client.4192.objecter shutdown clearing up homeless session... 2024-03-25T18:25:26.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.883+0000 7f5e78c93640 10 client.4192.objecter successfully canceled tick 2024-03-25T18:25:26.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.884+0000 7f5e78c93640 1 -- 172.21.15.3:0/2764134820 >> v1:172.21.15.3:6800/4195140966 conn(0x7f5e5406e100 legacy=0x7f5e540705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:26.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.884+0000 7f5e78c93640 10 monclient: shutdown 2024-03-25T18:25:26.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.884+0000 7f5e78c93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:26.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.884+0000 7f5e78c93640 1 -- 172.21.15.3:0/2764134820 >> v1:172.21.15.16:6789/0 conn(0x7f5e740b4e70 legacy=0x7f5e741e1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:26.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.884+0000 7f5e78c93640 1 -- 172.21.15.3:0/2764134820 shutdown_connections 2024-03-25T18:25:26.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.884+0000 7f5e78c93640 1 -- 172.21.15.3:0/2764134820 >> 172.21.15.3:0/2764134820 conn(0x7f5e740617a0 msgr2=0x7f5e740b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:26.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.884+0000 7f5e78c93640 1 -- 172.21.15.3:0/2764134820 shutdown_connections 2024-03-25T18:25:26.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.884+0000 7f5e78c93640 1 -- 172.21.15.3:0/2764134820 wait complete. 2024-03-25T18:25:26.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:26.884+0000 7f5e78c93640 1 librados: shutdown 2024-03-25T18:25:26.896 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:27.137 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:17.678 35490== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.648 INFO:tasks.ceph.osd.5.smithi016.stderr:==00:00:00:22.179 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.652 INFO:tasks.ceph.osd.5.smithi016.stderr:==00:00:00:22.183 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.832 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:22.397 35484== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.836 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:22.402 35484== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.903 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:22.462 35485== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.910 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:22.468 35485== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.065 INFO:tasks.ceph.osd.4.smithi016.stderr:==00:00:00:22.573 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.069 INFO:tasks.ceph.osd.4.smithi016.stderr:==00:00:00:22.577 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.234 INFO:tasks.ceph.osd.6.smithi016.stderr:==00:00:00:22.763 35188== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.238 INFO:tasks.ceph.osd.6.smithi016.stderr:==00:00:00:22.767 35188== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.374 INFO:tasks.ceph.osd.7.smithi016.stderr:==00:00:00:22.899 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.380 INFO:tasks.ceph.osd.7.smithi016.stderr:==00:00:00:22.905 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.441 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:23.009 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.447 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:23.015 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.622 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:23.163 35490== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.627 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:23.168 35490== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:33.703 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:33.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.799+0000 7f7d66425640 10 monclient: get_monmap_and_config 2024-03-25T18:25:33.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.799+0000 7f7d66425640 10 monclient: build_initial_monmap 2024-03-25T18:25:33.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.799+0000 7f7d66425640 10 monclient: monmap: 2024-03-25T18:25:33.800 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:25:33.800 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:33.800 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:25:33.800146+0000 2024-03-25T18:25:33.800 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:25:33.800146+0000 2024-03-25T18:25:33.800 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:33.801 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:33.801 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:33.801 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:25:33.801 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:25:33.801 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:33.801 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.800+0000 7f7d66425640 1 Processor -- start 2024-03-25T18:25:33.801 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.800+0000 7f7d66425640 1 -- start start 2024-03-25T18:25:33.801 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.800+0000 7f7d66425640 10 monclient: init 2024-03-25T18:25:33.803 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.801+0000 7f7d66425640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:33.803 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.801+0000 7f7d66425640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:33.803 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.801+0000 7f7d66425640 10 monclient(hunting): picked mon.noname-a con 0x7f7d600b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:33.803 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.801+0000 7f7d66425640 10 monclient(hunting): picked mon.noname-b con 0x7f7d600b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:33.803 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.801+0000 7f7d66425640 10 monclient(hunting): picked mon.noname-c con 0x7f7d6013c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:33.803 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.801+0000 7f7d66425640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d600ab0e0 con 0x7f7d600b4ff0 2024-03-25T18:25:33.803 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.801+0000 7f7d66425640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d600ab3c0 con 0x7f7d6013c480 2024-03-25T18:25:33.804 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.801+0000 7f7d66425640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d600a9f30 con 0x7f7d600b4670 2024-03-25T18:25:33.804 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.801+0000 7f7d66425640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:33.804 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.801+0000 7f7d66425640 10 monclient(hunting): authenticate will time out at 7388.061823s 2024-03-25T18:25:33.804 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.802+0000 7f7d649c2640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7d6013c480 0x7f7d6014dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:45910/0 (socket says 172.21.15.3:45910) 2024-03-25T18:25:33.804 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.802+0000 7f7d649c2640 1 -- 172.21.15.3:0/3287686075 learned_addr learned my addr 172.21.15.3:0/3287686075 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:25:33.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.806+0000 7f7d5effd640 1 -- 172.21.15.3:0/3287686075 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855856496 0 0) 0x7f7d600ab3c0 con 0x7f7d6013c480 2024-03-25T18:25:33.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.806+0000 7f7d5effd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:33.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.806+0000 7f7d5effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:33.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.806+0000 7f7d5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:33.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.806+0000 7f7d5effd640 10 monclient(hunting): my global_id is 4195 2024-03-25T18:25:33.808 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.806+0000 7f7d5effd640 1 -- 172.21.15.3:0/3287686075 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d48003520 con 0x7f7d6013c480 2024-03-25T18:25:33.808 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.807+0000 7f7d5effd640 1 -- 172.21.15.3:0/3287686075 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 170199709 0 0) 0x7f7d600a9f30 con 0x7f7d600b4670 2024-03-25T18:25:33.808 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.807+0000 7f7d5effd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:33.808 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.807+0000 7f7d5effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:33.808 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.807+0000 7f7d5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:33.808 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.807+0000 7f7d5effd640 10 monclient(hunting): my global_id is 4202 2024-03-25T18:25:33.808 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.807+0000 7f7d5effd640 1 -- 172.21.15.3:0/3287686075 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d600ab3c0 con 0x7f7d600b4670 2024-03-25T18:25:33.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.810+0000 7f7d5effd640 1 -- 172.21.15.3:0/3287686075 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 361720039 0 0) 0x7f7d48003520 con 0x7f7d6013c480 2024-03-25T18:25:33.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.810+0000 7f7d5effd640 1 -- 172.21.15.3:0/3287686075 >> v1:172.21.15.3:6790/0 conn(0x7f7d600b4670 legacy=0x7f7d601425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:33.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.811+0000 7f7d5effd640 1 -- 172.21.15.3:0/3287686075 >> v1:172.21.15.3:6789/0 conn(0x7f7d600b4ff0 legacy=0x7f7d6013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:33.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.811+0000 7f7d5effd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:33.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.811+0000 7f7d5effd640 1 monclient: found mon.noname-c 2024-03-25T18:25:33.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.811+0000 7f7d5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:33.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.811+0000 7f7d5effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:25:33.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.811+0000 7f7d5effd640 1 -- 172.21.15.3:0/3287686075 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d600a9b70 con 0x7f7d6013c480 2024-03-25T18:25:33.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.811+0000 7f7d5effd640 10 monclient: _finish_auth 0 2024-03-25T18:25:33.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.811+0000 7f7d5effd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:33.813 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.811+0000 7f7d5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:33.813 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.811+0000 7f7d66425640 5 monclient: authenticate success, global_id 4195 2024-03-25T18:25:33.813 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.811+0000 7f7d66425640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:33.817 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.813+0000 7f7d5effd640 1 -- 172.21.15.3:0/3287686075 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7d500019d0 con 0x7f7d6013c480 2024-03-25T18:25:33.817 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.813+0000 7f7d5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:33.817 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.813+0000 7f7d5effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:33.817 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.813+0000 7f7d5effd640 10 monclient: dump: 2024-03-25T18:25:33.817 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:33.817 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:33.817 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:33.817 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:33.817 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:33.818 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:33.818 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:33.818 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:33.818 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:33.818 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:33.818 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.813+0000 7f7d5effd640 1 -- 172.21.15.3:0/3287686075 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d50001c20 con 0x7f7d6013c480 2024-03-25T18:25:33.818 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.813+0000 7f7d66425640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:33.818 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.813+0000 7f7d5effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:33.818 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.813+0000 7f7d5effd640 1 -- 172.21.15.3:0/3287686075 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7d50001ff0 con 0x7f7d6013c480 2024-03-25T18:25:33.818 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.813+0000 7f7d5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:33.818 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.813+0000 7f7d5effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:33.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.813+0000 7f7d5effd640 10 monclient: dump: 2024-03-25T18:25:33.819 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:33.819 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:33.819 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:33.819 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:33.819 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:33.819 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:33.819 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:33.819 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:33.819 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:33.819 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:33.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.813+0000 7f7d66425640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:33.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.813+0000 7f7d66425640 4 set_mon_vals no callback set 2024-03-25T18:25:33.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.814+0000 7f7d66425640 10 monclient: shutdown 2024-03-25T18:25:33.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.814+0000 7f7d66425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:33.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.814+0000 7f7d66425640 1 -- 172.21.15.3:0/3287686075 >> v1:172.21.15.16:6789/0 conn(0x7f7d6013c480 legacy=0x7f7d6014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:33.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.814+0000 7f7d66425640 1 -- 172.21.15.3:0/3287686075 shutdown_connections 2024-03-25T18:25:33.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.814+0000 7f7d66425640 1 -- 172.21.15.3:0/3287686075 >> 172.21.15.3:0/3287686075 conn(0x7f7d600617a0 msgr2=0x7f7d60061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:33.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.814+0000 7f7d66425640 1 -- 172.21.15.3:0/3287686075 shutdown_connections 2024-03-25T18:25:33.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.814+0000 7f7d66425640 1 -- 172.21.15.3:0/3287686075 wait complete. 2024-03-25T18:25:33.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.815+0000 7f7d66425640 10 monclient: build_initial_monmap 2024-03-25T18:25:33.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.815+0000 7f7d66425640 10 monclient: monmap: 2024-03-25T18:25:33.820 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:25:33.820 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:33.821 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:25:33.821 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:25:33.821 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:33.821 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:25:33.821 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:33.821 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:25:33.821 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:25:33.821 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:33.821 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.816+0000 7f7d66425640 1 librados: starting msgr at 2024-03-25T18:25:33.821 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.816+0000 7f7d66425640 1 librados: starting objecter 2024-03-25T18:25:33.821 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.816+0000 7f7d66425640 1 Processor -- start 2024-03-25T18:25:33.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.816+0000 7f7d66425640 1 -- start start 2024-03-25T18:25:33.826 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.816+0000 7f7d66425640 1 librados: setting wanted keys 2024-03-25T18:25:33.826 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.816+0000 7f7d66425640 1 librados: calling monclient init 2024-03-25T18:25:33.826 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.816+0000 7f7d66425640 10 monclient: init 2024-03-25T18:25:33.826 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.817+0000 7f7d66425640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:33.826 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.817+0000 7f7d66425640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:33.826 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.817+0000 7f7d66425640 10 monclient(hunting): picked mon.noname-a con 0x7f7d600b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:33.826 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.817+0000 7f7d66425640 10 monclient(hunting): picked mon.noname-c con 0x7f7d600b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:33.826 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.817+0000 7f7d66425640 10 monclient(hunting): picked mon.noname-b con 0x7f7d6013c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:33.826 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.818+0000 7f7d66425640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d600abfd0 con 0x7f7d600b4670 2024-03-25T18:25:33.826 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.818+0000 7f7d66425640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d600ac2b0 con 0x7f7d6013c480 2024-03-25T18:25:33.826 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.818+0000 7f7d66425640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d600ac590 con 0x7f7d600b4ff0 2024-03-25T18:25:33.826 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.818+0000 7f7d66425640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:33.826 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.818+0000 7f7d66425640 10 monclient(hunting): authenticate will time out at 7388.078535s 2024-03-25T18:25:33.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.818+0000 7f7d5ffff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f7d600b4670 0x7f7d6013b440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59080/0 (socket says 172.21.15.3:59080) 2024-03-25T18:25:33.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.818+0000 7f7d649c2640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7d6013c480 0x7f7d60136890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:45920/0 (socket says 172.21.15.3:45920) 2024-03-25T18:25:33.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.818+0000 7f7d5f7fe640 1 --1- 172.21.15.3:0/4137851210 >> v1:172.21.15.3:6790/0 conn(0x7f7d600b4ff0 0x7f7d60136180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:47076/0 (socket says 172.21.15.3:47076) 2024-03-25T18:25:33.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.818+0000 7f7d5ffff640 1 -- 172.21.15.3:0/4137851210 learned_addr learned my addr 172.21.15.3:0/4137851210 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:25:33.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.819+0000 7f7d5cff9640 10 client.?.objecter ms_handle_connect 0x7f7d6013c480 2024-03-25T18:25:33.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.819+0000 7f7d5cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:33.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.819+0000 7f7d5cff9640 10 client.?.objecter ms_handle_connect 0x7f7d600b4670 2024-03-25T18:25:33.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.819+0000 7f7d5cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:33.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.819+0000 7f7d5cff9640 10 client.?.objecter ms_handle_connect 0x7f7d600b4ff0 2024-03-25T18:25:33.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.819+0000 7f7d5cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:33.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.821+0000 7f7d5cff9640 1 -- 172.21.15.3:0/4137851210 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3515402462 0 0) 0x7f7d600ac2b0 con 0x7f7d6013c480 2024-03-25T18:25:33.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.821+0000 7f7d5cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:33.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.821+0000 7f7d5cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:33.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.821+0000 7f7d5cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:33.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.821+0000 7f7d5cff9640 10 monclient(hunting): my global_id is 4198 2024-03-25T18:25:33.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.821+0000 7f7d5cff9640 1 -- 172.21.15.3:0/4137851210 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d4c004eb0 con 0x7f7d6013c480 2024-03-25T18:25:33.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.822+0000 7f7d5cff9640 1 -- 172.21.15.3:0/4137851210 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1719325816 0 0) 0x7f7d600abfd0 con 0x7f7d600b4670 2024-03-25T18:25:33.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.822+0000 7f7d5cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:33.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.822+0000 7f7d5cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:33.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.822+0000 7f7d5cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:33.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.822+0000 7f7d5cff9640 10 monclient(hunting): my global_id is 4233 2024-03-25T18:25:33.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.822+0000 7f7d5cff9640 1 -- 172.21.15.3:0/4137851210 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d600ac2b0 con 0x7f7d600b4670 2024-03-25T18:25:33.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.822+0000 7f7d5cff9640 1 -- 172.21.15.3:0/4137851210 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 537713321 0 0) 0x7f7d600ac590 con 0x7f7d600b4ff0 2024-03-25T18:25:33.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.822+0000 7f7d5cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:33.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.822+0000 7f7d5cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:33.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.822+0000 7f7d5cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:33.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.822+0000 7f7d5cff9640 10 monclient(hunting): my global_id is 4205 2024-03-25T18:25:33.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.822+0000 7f7d5cff9640 1 -- 172.21.15.3:0/4137851210 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d4c005360 con 0x7f7d600b4ff0 2024-03-25T18:25:33.829 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.828+0000 7f7d5cff9640 1 -- 172.21.15.3:0/4137851210 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2096903564 0 0) 0x7f7d600ac2b0 con 0x7f7d600b4670 2024-03-25T18:25:33.829 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.828+0000 7f7d5cff9640 1 -- 172.21.15.3:0/4137851210 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d4c0057b0 con 0x7f7d600b4670 2024-03-25T18:25:33.829 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.828+0000 7f7d5cff9640 1 -- 172.21.15.3:0/4137851210 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2323588163 0 0) 0x7f7d4c004eb0 con 0x7f7d6013c480 2024-03-25T18:25:33.829 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.828+0000 7f7d5cff9640 1 -- 172.21.15.3:0/4137851210 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d4c005d60 con 0x7f7d6013c480 2024-03-25T18:25:33.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.829+0000 7f7d5cff9640 1 -- 172.21.15.3:0/4137851210 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7d50001f70 con 0x7f7d6013c480 2024-03-25T18:25:33.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.829+0000 7f7d5cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:33.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.829+0000 7f7d5cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:33.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.829+0000 7f7d5cff9640 10 monclient(hunting): dump: 2024-03-25T18:25:33.830 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:33.830 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:33.830 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:33.830 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:33.831 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:33.831 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:33.831 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:33.831 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:33.831 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:33.831 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:33.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.829+0000 7f7d5cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:33.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.829+0000 7f7d5cff9640 1 -- 172.21.15.3:0/4137851210 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7d580019d0 con 0x7f7d600b4670 2024-03-25T18:25:33.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.829+0000 7f7d5cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:33.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.829+0000 7f7d5cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:33.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.829+0000 7f7d5cff9640 10 monclient(hunting): dump: 2024-03-25T18:25:33.832 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:33.832 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:33.832 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:33.832 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:33.832 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:33.832 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:33.832 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:33.833 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:33.833 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:33.833 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:33.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.831+0000 7f7d5cff9640 1 -- 172.21.15.3:0/4137851210 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1836331024 0 0) 0x7f7d4c005360 con 0x7f7d600b4ff0 2024-03-25T18:25:33.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.831+0000 7f7d5cff9640 1 -- 172.21.15.3:0/4137851210 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d4c004eb0 con 0x7f7d600b4ff0 2024-03-25T18:25:33.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.832+0000 7f7d5cff9640 1 -- 172.21.15.3:0/4137851210 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7d54001850 con 0x7f7d600b4ff0 2024-03-25T18:25:33.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.832+0000 7f7d5cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:33.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.832+0000 7f7d5cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:33.834 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.832+0000 7f7d5cff9640 10 monclient(hunting): dump: 2024-03-25T18:25:33.834 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:33.834 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:33.834 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:33.834 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:33.834 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:33.834 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:33.834 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:33.834 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:33.834 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:33.834 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:33.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.834+0000 7f7d5cff9640 1 -- 172.21.15.3:0/4137851210 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3508000418 0 0) 0x7f7d4c005d60 con 0x7f7d6013c480 2024-03-25T18:25:33.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.834+0000 7f7d5cff9640 1 -- 172.21.15.3:0/4137851210 >> v1:172.21.15.3:6790/0 conn(0x7f7d600b4ff0 legacy=0x7f7d60136180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:33.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.834+0000 7f7d5cff9640 1 -- 172.21.15.3:0/4137851210 >> v1:172.21.15.3:6789/0 conn(0x7f7d600b4670 legacy=0x7f7d6013b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:33.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.834+0000 7f7d5cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:25:33.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.834+0000 7f7d5cff9640 1 monclient: found mon.b 2024-03-25T18:25:33.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.834+0000 7f7d5cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:33.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.834+0000 7f7d5cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:33.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.834+0000 7f7d5cff9640 1 -- 172.21.15.3:0/4137851210 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d6013bf50 con 0x7f7d6013c480 2024-03-25T18:25:33.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.834+0000 7f7d5cff9640 10 monclient: _finish_auth 0 2024-03-25T18:25:33.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.834+0000 7f7d5cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:25:33.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.834+0000 7f7d5cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:33.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.834+0000 7f7d66425640 5 monclient: authenticate success, global_id 4198 2024-03-25T18:25:33.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.834+0000 7f7d66425640 10 monclient: _renew_subs 2024-03-25T18:25:33.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.834+0000 7f7d66425640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:33.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.834+0000 7f7d66425640 1 -- 172.21.15.3:0/4137851210 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d6013bb50 con 0x7f7d6013c480 2024-03-25T18:25:33.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.834+0000 7f7d66425640 10 client.4198.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:33.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.834+0000 7f7d66425640 10 monclient: _renew_subs 2024-03-25T18:25:33.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.834+0000 7f7d66425640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:33.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.834+0000 7f7d66425640 1 -- 172.21.15.3:0/4137851210 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d601e38d0 con 0x7f7d6013c480 2024-03-25T18:25:33.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.834+0000 7f7d66425640 1 librados: init done 2024-03-25T18:25:33.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.835+0000 7f7d66425640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:33.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.835+0000 7f7d66425640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:33.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.835+0000 7f7d66425640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:33.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:33.835+0000 7f7d66425640 1 -- 172.21.15.3:0/4137851210 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d600688c0 con 0x7f7d6013c480 2024-03-25T18:25:34.195 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.194+0000 7f7d5cff9640 1 -- 172.21.15.3:0/4137851210 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d500020d0 con 0x7f7d6013c480 2024-03-25T18:25:34.195 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.194+0000 7f7d5cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:34.195 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.194+0000 7f7d5cff9640 1 -- 172.21.15.3:0/4137851210 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7d50004fb0 con 0x7f7d6013c480 2024-03-25T18:25:34.195 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.194+0000 7f7d5cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:34.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.194+0000 7f7d5cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:34.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.194+0000 7f7d5cff9640 10 monclient: dump: 2024-03-25T18:25:34.196 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:34.196 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:34.196 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:34.197 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:34.197 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:34.197 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:34.197 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:34.197 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:34.197 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:34.198 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:34.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.194+0000 7f7d5e7fc640 4 set_mon_vals no callback set 2024-03-25T18:25:34.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.194+0000 7f7d5cff9640 1 -- 172.21.15.3:0/4137851210 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f7d5001af80 con 0x7f7d6013c480 2024-03-25T18:25:34.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.195+0000 7f7d5ffff640 10 monclient: get_auth_request con 0x7f7d4c06e9b0 auth_method 0 2024-03-25T18:25:34.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.195+0000 7f7d5cff9640 10 client.4198.objecter ms_handle_connect 0x7f7d4c06e9b0 2024-03-25T18:25:34.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.202+0000 7f7d5cff9640 1 -- 172.21.15.3:0/4137851210 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f7d50054ae0 con 0x7f7d6013c480 2024-03-25T18:25:34.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.202+0000 7f7d5cff9640 10 client.4198.objecter ms_dispatch 0x7f7d60000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:34.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.202+0000 7f7d5cff9640 3 client.4198.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:34.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.202+0000 7f7d5cff9640 3 client.4198.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:34.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.202+0000 7f7d5cff9640 20 client.4198.objecter dump_active .. 0 homeless 2024-03-25T18:25:34.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.203+0000 7f7d5cff9640 1 -- 172.21.15.3:0/4137851210 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7d500505f0 con 0x7f7d6013c480 2024-03-25T18:25:34.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.203+0000 7f7d5cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:34.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.203+0000 7f7d5cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:34.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.428+0000 7f7d66425640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:34.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.428+0000 7f7d66425640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:34.430 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.428+0000 7f7d66425640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:34.430 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.428+0000 7f7d66425640 1 -- 172.21.15.3:0/4137851210 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d6006dbd0 con 0x7f7d6013c480 2024-03-25T18:25:34.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.444+0000 7f7d5cff9640 1 -- 172.21.15.3:0/4137851210 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f7d50052520 con 0x7f7d6013c480 2024-03-25T18:25:34.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.444+0000 7f7d5cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:34.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.444+0000 7f7d5cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:34.446 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:25:34.446 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:34.447 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.446+0000 7f7d66425640 10 librados: watch_flush enter 2024-03-25T18:25:34.447 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.446+0000 7f7d66425640 10 librados: watch_flush exit 2024-03-25T18:25:34.448 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.446+0000 7f7d66425640 20 client.4198.objecter shutdown clearing up homeless session... 2024-03-25T18:25:34.448 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.446+0000 7f7d66425640 10 client.4198.objecter successfully canceled tick 2024-03-25T18:25:34.448 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.446+0000 7f7d66425640 1 -- 172.21.15.3:0/4137851210 >> v1:172.21.15.3:6800/4195140966 conn(0x7f7d4c06e9b0 legacy=0x7f7d4c070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:34.448 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.446+0000 7f7d66425640 10 monclient: shutdown 2024-03-25T18:25:34.448 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.446+0000 7f7d66425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:34.448 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.446+0000 7f7d66425640 1 -- 172.21.15.3:0/4137851210 >> v1:172.21.15.16:6789/0 conn(0x7f7d6013c480 legacy=0x7f7d60136890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:34.448 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.446+0000 7f7d66425640 1 -- 172.21.15.3:0/4137851210 shutdown_connections 2024-03-25T18:25:34.448 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.446+0000 7f7d66425640 1 -- 172.21.15.3:0/4137851210 >> 172.21.15.3:0/4137851210 conn(0x7f7d600617a0 msgr2=0x7f7d60140760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:34.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.447+0000 7f7d66425640 1 -- 172.21.15.3:0/4137851210 shutdown_connections 2024-03-25T18:25:34.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.447+0000 7f7d66425640 1 -- 172.21.15.3:0/4137851210 wait complete. 2024-03-25T18:25:34.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:34.447+0000 7f7d66425640 1 librados: shutdown 2024-03-25T18:25:34.459 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:36.319 INFO:tasks.ceph.osd.5.smithi016.stderr:==00:00:00:26.850 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:36.502 INFO:tasks.ceph.osd.4.smithi016.stderr:==00:00:00:27.009 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:36.522 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:00:27.087 35484== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:36.543 INFO:tasks.ceph.osd.7.smithi016.stderr:==00:00:00:27.068 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:36.704 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:00:27.263 35485== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:36.824 INFO:tasks.ceph.osd.6.smithi016.stderr:==00:00:00:27.353 35188== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:37.016 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:00:27.583 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:37.284 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:00:27.825 35490== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:41.265 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:41.355 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.354+0000 7f00fc88f640 10 monclient: get_monmap_and_config 2024-03-25T18:25:41.355 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.354+0000 7f00fc88f640 10 monclient: build_initial_monmap 2024-03-25T18:25:41.355 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.354+0000 7f00fc88f640 10 monclient: monmap: 2024-03-25T18:25:41.355 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:25:41.355 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:41.355 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:25:41.355478+0000 2024-03-25T18:25:41.356 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:25:41.355478+0000 2024-03-25T18:25:41.356 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:41.356 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:41.356 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:41.356 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:25:41.356 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:25:41.356 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:41.357 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.355+0000 7f00fc88f640 1 Processor -- start 2024-03-25T18:25:41.357 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.355+0000 7f00fc88f640 1 -- start start 2024-03-25T18:25:41.357 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.355+0000 7f00fc88f640 10 monclient: init 2024-03-25T18:25:41.357 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.355+0000 7f00fc88f640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:41.357 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.355+0000 7f00fc88f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:41.357 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.355+0000 7f00fc88f640 10 monclient(hunting): picked mon.noname-a con 0x7f00f80b4e70 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:41.357 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.355+0000 7f00fc88f640 10 monclient(hunting): picked mon.noname-c con 0x7f00f80b44b0 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:41.357 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.355+0000 7f00fc88f640 10 monclient(hunting): picked mon.noname-b con 0x7f00f80b1b00 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:41.357 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.355+0000 7f00fc88f640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f80a96f0 con 0x7f00f80b4e70 2024-03-25T18:25:41.358 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.355+0000 7f00fc88f640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f80ab0e0 con 0x7f00f80b44b0 2024-03-25T18:25:41.358 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.355+0000 7f00fc88f640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f8145f70 con 0x7f00f80b1b00 2024-03-25T18:25:41.358 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.355+0000 7f00fc88f640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:41.358 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.355+0000 7f00fc88f640 10 monclient(hunting): authenticate will time out at 7395.616315s 2024-03-25T18:25:41.358 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.356+0000 7f00effff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f00f80b44b0 0x7f00f8145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:56006/0 (socket says 172.21.15.3:56006) 2024-03-25T18:25:41.358 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.356+0000 7f00effff640 1 -- 172.21.15.3:0/4227317069 learned_addr learned my addr 172.21.15.3:0/4227317069 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:25:41.360 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.359+0000 7f00f5d9b640 1 -- 172.21.15.3:0/4227317069 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2524459004 0 0) 0x7f00f8145f70 con 0x7f00f80b1b00 2024-03-25T18:25:41.360 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.359+0000 7f00f5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:25:41.360 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.359+0000 7f00f5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:41.360 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.359+0000 7f00f5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:41.361 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.359+0000 7f00f5d9b640 10 monclient(hunting): my global_id is 4208 2024-03-25T18:25:41.361 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.359+0000 7f00f5d9b640 1 -- 172.21.15.3:0/4227317069 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00e4003520 con 0x7f00f80b1b00 2024-03-25T18:25:41.361 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.359+0000 7f00f5d9b640 1 -- 172.21.15.3:0/4227317069 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1636642430 0 0) 0x7f00f80ab0e0 con 0x7f00f80b44b0 2024-03-25T18:25:41.361 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.359+0000 7f00f5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:25:41.361 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.359+0000 7f00f5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:41.361 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.359+0000 7f00f5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:41.361 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.359+0000 7f00f5d9b640 10 monclient(hunting): my global_id is 4201 2024-03-25T18:25:41.361 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.359+0000 7f00f5d9b640 1 -- 172.21.15.3:0/4227317069 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00f8145f70 con 0x7f00f80b44b0 2024-03-25T18:25:41.361 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.360+0000 7f00f5d9b640 1 -- 172.21.15.3:0/4227317069 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3386596975 0 0) 0x7f00f80a96f0 con 0x7f00f80b4e70 2024-03-25T18:25:41.362 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.360+0000 7f00f5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:25:41.362 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.360+0000 7f00f5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:41.362 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.360+0000 7f00f5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:41.362 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.360+0000 7f00f5d9b640 10 monclient(hunting): my global_id is 4236 2024-03-25T18:25:41.362 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.360+0000 7f00f5d9b640 1 -- 172.21.15.3:0/4227317069 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00f80ab0e0 con 0x7f00f80b4e70 2024-03-25T18:25:41.365 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.364+0000 7f00f5d9b640 1 -- 172.21.15.3:0/4227317069 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 262440120 0 0) 0x7f00e4003520 con 0x7f00f80b1b00 2024-03-25T18:25:41.365 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.364+0000 7f00f5d9b640 1 -- 172.21.15.3:0/4227317069 >> v1:172.21.15.16:6789/0 conn(0x7f00f80b44b0 legacy=0x7f00f8145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:41.365 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.364+0000 7f00f5d9b640 1 -- 172.21.15.3:0/4227317069 >> v1:172.21.15.3:6789/0 conn(0x7f00f80b4e70 legacy=0x7f00f813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:41.366 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.364+0000 7f00f5d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:25:41.366 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.364+0000 7f00f5d9b640 1 monclient: found mon.noname-b 2024-03-25T18:25:41.366 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.364+0000 7f00f5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:41.366 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.364+0000 7f00f5d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-25T18:25:41.366 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.364+0000 7f00f5d9b640 1 -- 172.21.15.3:0/4227317069 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f80a9b80 con 0x7f00f80b1b00 2024-03-25T18:25:41.366 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.364+0000 7f00f5d9b640 10 monclient: _finish_auth 0 2024-03-25T18:25:41.366 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.364+0000 7f00f5d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:25:41.366 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.364+0000 7f00f5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:41.366 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.364+0000 7f00f5d9b640 1 -- 172.21.15.3:0/4227317069 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f00f0001a10 con 0x7f00f80b1b00 2024-03-25T18:25:41.366 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.364+0000 7f00f5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:41.366 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.364+0000 7f00f5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:41.366 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.364+0000 7f00f5d9b640 10 monclient: dump: 2024-03-25T18:25:41.366 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:41.367 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:41.367 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:41.367 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:41.367 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:41.367 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:41.367 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:41.367 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:41.367 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:41.367 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:41.367 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.364+0000 7f00fc88f640 5 monclient: authenticate success, global_id 4208 2024-03-25T18:25:41.367 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.364+0000 7f00fc88f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:41.369 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.366+0000 7f00f5d9b640 1 -- 172.21.15.3:0/4227317069 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00f0001b70 con 0x7f00f80b1b00 2024-03-25T18:25:41.369 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.366+0000 7f00f5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:41.369 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.366+0000 7f00f5d9b640 1 -- 172.21.15.3:0/4227317069 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f00f0001f80 con 0x7f00f80b1b00 2024-03-25T18:25:41.369 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.366+0000 7f00f5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:41.369 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.366+0000 7f00f5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:41.369 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.366+0000 7f00f5d9b640 10 monclient: dump: 2024-03-25T18:25:41.369 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:41.370 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:41.370 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:41.370 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:41.370 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:41.370 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:41.370 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:41.370 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:41.370 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:41.370 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:41.370 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.366+0000 7f00fc88f640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:41.370 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.366+0000 7f00fc88f640 4 set_mon_vals no callback set 2024-03-25T18:25:41.370 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.367+0000 7f00fc88f640 10 monclient: shutdown 2024-03-25T18:25:41.370 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.367+0000 7f00fc88f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:41.371 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.367+0000 7f00fc88f640 1 -- 172.21.15.3:0/4227317069 >> v1:172.21.15.3:6790/0 conn(0x7f00f80b1b00 legacy=0x7f00f814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:41.371 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.367+0000 7f00fc88f640 1 -- 172.21.15.3:0/4227317069 shutdown_connections 2024-03-25T18:25:41.371 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.367+0000 7f00fc88f640 1 -- 172.21.15.3:0/4227317069 >> 172.21.15.3:0/4227317069 conn(0x7f00f80617a0 msgr2=0x7f00f8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:41.371 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.367+0000 7f00fc88f640 1 -- 172.21.15.3:0/4227317069 shutdown_connections 2024-03-25T18:25:41.371 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.367+0000 7f00fc88f640 1 -- 172.21.15.3:0/4227317069 wait complete. 2024-03-25T18:25:41.374 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.368+0000 7f00fc88f640 10 monclient: build_initial_monmap 2024-03-25T18:25:41.374 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.368+0000 7f00fc88f640 10 monclient: monmap: 2024-03-25T18:25:41.374 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:25:41.374 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:41.374 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:25:41.374 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:25:41.374 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:41.375 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:25:41.375 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:41.375 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:25:41.375 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:25:41.375 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:41.375 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.369+0000 7f00fc88f640 1 librados: starting msgr at 2024-03-25T18:25:41.375 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.369+0000 7f00fc88f640 1 librados: starting objecter 2024-03-25T18:25:41.375 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.369+0000 7f00fc88f640 1 Processor -- start 2024-03-25T18:25:41.375 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.369+0000 7f00fc88f640 1 -- start start 2024-03-25T18:25:41.375 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.369+0000 7f00fc88f640 1 librados: setting wanted keys 2024-03-25T18:25:41.375 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.369+0000 7f00fc88f640 1 librados: calling monclient init 2024-03-25T18:25:41.375 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.369+0000 7f00fc88f640 10 monclient: init 2024-03-25T18:25:41.375 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.369+0000 7f00fc88f640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:41.375 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.370+0000 7f00fc88f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:41.376 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.370+0000 7f00fc88f640 10 monclient(hunting): picked mon.noname-a con 0x7f00f80b1b00 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:41.376 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.370+0000 7f00fc88f640 10 monclient(hunting): picked mon.noname-c con 0x7f00f80b44b0 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:41.376 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.370+0000 7f00fc88f640 10 monclient(hunting): picked mon.noname-b con 0x7f00f80b4e70 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:41.376 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.370+0000 7f00fc88f640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f81df1c0 con 0x7f00f80b1b00 2024-03-25T18:25:41.376 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.370+0000 7f00fc88f640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f8141b30 con 0x7f00f80b4e70 2024-03-25T18:25:41.376 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.370+0000 7f00fc88f640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f8141e10 con 0x7f00f80b44b0 2024-03-25T18:25:41.376 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.370+0000 7f00fc88f640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:41.376 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.370+0000 7f00fc88f640 10 monclient(hunting): authenticate will time out at 7395.630952s 2024-03-25T18:25:41.376 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.370+0000 7f00f659c640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f00f80b1b00 0x7f00f813c840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:47334/0 (socket says 172.21.15.3:47334) 2024-03-25T18:25:41.376 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.370+0000 7f00f659c640 1 -- 172.21.15.3:0/3556230185 learned_addr learned my addr 172.21.15.3:0/3556230185 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:25:41.376 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.371+0000 7f00eeffd640 10 client.?.objecter ms_handle_connect 0x7f00f80b1b00 2024-03-25T18:25:41.376 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.371+0000 7f00eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:41.376 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.371+0000 7f00eeffd640 10 client.?.objecter ms_handle_connect 0x7f00f80b4e70 2024-03-25T18:25:41.376 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.371+0000 7f00eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:41.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.371+0000 7f00eeffd640 10 client.?.objecter ms_handle_connect 0x7f00f80b44b0 2024-03-25T18:25:41.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.371+0000 7f00eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:41.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.374+0000 7f00eeffd640 1 -- 172.21.15.3:0/3556230185 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198385017 0 0) 0x7f00f8141b30 con 0x7f00f80b4e70 2024-03-25T18:25:41.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.374+0000 7f00eeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:41.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.374+0000 7f00eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:41.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.374+0000 7f00eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:41.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.374+0000 7f00eeffd640 10 monclient(hunting): my global_id is 4204 2024-03-25T18:25:41.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.374+0000 7f00eeffd640 1 -- 172.21.15.3:0/3556230185 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00d8004eb0 con 0x7f00f80b4e70 2024-03-25T18:25:41.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.374+0000 7f00eeffd640 1 -- 172.21.15.3:0/3556230185 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1187283419 0 0) 0x7f00f81df1c0 con 0x7f00f80b1b00 2024-03-25T18:25:41.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.374+0000 7f00eeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:41.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.374+0000 7f00eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:41.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.374+0000 7f00eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:41.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.374+0000 7f00eeffd640 10 monclient(hunting): my global_id is 4239 2024-03-25T18:25:41.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.374+0000 7f00eeffd640 1 -- 172.21.15.3:0/3556230185 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00f8141b30 con 0x7f00f80b1b00 2024-03-25T18:25:41.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.374+0000 7f00eeffd640 1 -- 172.21.15.3:0/3556230185 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4064961303 0 0) 0x7f00f8141e10 con 0x7f00f80b44b0 2024-03-25T18:25:41.379 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.374+0000 7f00eeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:41.379 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.374+0000 7f00eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:41.379 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.374+0000 7f00eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:41.379 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.374+0000 7f00eeffd640 10 monclient(hunting): my global_id is 4211 2024-03-25T18:25:41.379 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.374+0000 7f00eeffd640 1 -- 172.21.15.3:0/3556230185 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00f81df1c0 con 0x7f00f80b44b0 2024-03-25T18:25:41.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.381+0000 7f00eeffd640 1 -- 172.21.15.3:0/3556230185 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1824487034 0 0) 0x7f00d8004eb0 con 0x7f00f80b4e70 2024-03-25T18:25:41.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.381+0000 7f00eeffd640 1 -- 172.21.15.3:0/3556230185 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00f8141e10 con 0x7f00f80b4e70 2024-03-25T18:25:41.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.381+0000 7f00eeffd640 1 -- 172.21.15.3:0/3556230185 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f00f0001e80 con 0x7f00f80b4e70 2024-03-25T18:25:41.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.381+0000 7f00eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:41.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.381+0000 7f00eeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:41.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.381+0000 7f00eeffd640 10 monclient(hunting): dump: 2024-03-25T18:25:41.383 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:41.383 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:41.383 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:41.383 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:41.383 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:41.383 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:41.384 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:41.384 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:41.384 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:41.384 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:41.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.381+0000 7f00eeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:41.385 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.381+0000 7f00eeffd640 1 -- 172.21.15.3:0/3556230185 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1728113139 0 0) 0x7f00f81df1c0 con 0x7f00f80b44b0 2024-03-25T18:25:41.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.381+0000 7f00eeffd640 1 -- 172.21.15.3:0/3556230185 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00d8004eb0 con 0x7f00f80b44b0 2024-03-25T18:25:41.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.381+0000 7f00eeffd640 1 -- 172.21.15.3:0/3556230185 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f00e0001930 con 0x7f00f80b44b0 2024-03-25T18:25:41.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.381+0000 7f00eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:41.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.381+0000 7f00eeffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:41.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.381+0000 7f00eeffd640 10 monclient(hunting): dump: 2024-03-25T18:25:41.386 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:41.387 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:41.387 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:41.387 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:41.387 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:41.387 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:41.387 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:41.387 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:41.388 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:41.388 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:41.388 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.382+0000 7f00eeffd640 1 -- 172.21.15.3:0/3556230185 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1046456837 0 0) 0x7f00f8141b30 con 0x7f00f80b1b00 2024-03-25T18:25:41.388 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.382+0000 7f00eeffd640 1 -- 172.21.15.3:0/3556230185 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00f81df1c0 con 0x7f00f80b1b00 2024-03-25T18:25:41.388 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.382+0000 7f00eeffd640 1 -- 172.21.15.3:0/3556230185 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f00e80018d0 con 0x7f00f80b1b00 2024-03-25T18:25:41.388 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.382+0000 7f00eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:41.388 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.382+0000 7f00eeffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:41.388 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.382+0000 7f00eeffd640 10 monclient(hunting): dump: 2024-03-25T18:25:41.388 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:41.388 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:41.389 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:41.389 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:41.389 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:41.389 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:41.389 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:41.389 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:41.389 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:41.389 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:41.393 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.385+0000 7f00eeffd640 1 -- 172.21.15.3:0/3556230185 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2214796179 0 0) 0x7f00f8141e10 con 0x7f00f80b4e70 2024-03-25T18:25:41.393 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.385+0000 7f00eeffd640 1 -- 172.21.15.3:0/3556230185 >> v1:172.21.15.3:6790/0 conn(0x7f00f80b44b0 legacy=0x7f00f81df650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:41.393 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.385+0000 7f00eeffd640 1 -- 172.21.15.3:0/3556230185 >> v1:172.21.15.3:6789/0 conn(0x7f00f80b1b00 legacy=0x7f00f813c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:41.393 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.385+0000 7f00eeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:41.394 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.385+0000 7f00eeffd640 1 monclient: found mon.b 2024-03-25T18:25:41.394 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.385+0000 7f00eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:41.394 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.385+0000 7f00eeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:41.394 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.385+0000 7f00eeffd640 1 -- 172.21.15.3:0/3556230185 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f8142230 con 0x7f00f80b4e70 2024-03-25T18:25:41.394 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.385+0000 7f00eeffd640 10 monclient: _finish_auth 0 2024-03-25T18:25:41.394 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.385+0000 7f00eeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:41.394 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.385+0000 7f00eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:41.394 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.385+0000 7f00fc88f640 5 monclient: authenticate success, global_id 4204 2024-03-25T18:25:41.394 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.385+0000 7f00fc88f640 10 monclient: _renew_subs 2024-03-25T18:25:41.394 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.385+0000 7f00fc88f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:41.394 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.385+0000 7f00fc88f640 1 -- 172.21.15.3:0/3556230185 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00f81406a0 con 0x7f00f80b4e70 2024-03-25T18:25:41.394 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.385+0000 7f00fc88f640 10 client.4204.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:41.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.385+0000 7f00fc88f640 10 monclient: _renew_subs 2024-03-25T18:25:41.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.385+0000 7f00fc88f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:41.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.385+0000 7f00fc88f640 1 -- 172.21.15.3:0/3556230185 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f00f81dff40 con 0x7f00f80b4e70 2024-03-25T18:25:41.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.385+0000 7f00fc88f640 1 librados: init done 2024-03-25T18:25:41.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.387+0000 7f00fc88f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:41.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.387+0000 7f00fc88f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:41.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.387+0000 7f00fc88f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:41.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.387+0000 7f00fc88f640 1 -- 172.21.15.3:0/3556230185 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00f804f730 con 0x7f00f80b4e70 2024-03-25T18:25:41.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.649+0000 7f00eeffd640 1 -- 172.21.15.3:0/3556230185 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00f0002020 con 0x7f00f80b4e70 2024-03-25T18:25:41.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.649+0000 7f00eeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:41.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.649+0000 7f00eeffd640 1 -- 172.21.15.3:0/3556230185 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f00f0003dc0 con 0x7f00f80b4e70 2024-03-25T18:25:41.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.649+0000 7f00eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:41.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.649+0000 7f00eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:41.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.649+0000 7f00f559a640 4 set_mon_vals no callback set 2024-03-25T18:25:41.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.649+0000 7f00eeffd640 10 monclient: dump: 2024-03-25T18:25:41.651 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:41.651 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:41.651 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:41.651 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:41.651 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:41.651 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:41.652 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:41.652 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:41.652 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:41.652 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:41.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.650+0000 7f00eeffd640 1 -- 172.21.15.3:0/3556230185 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f00f0019d90 con 0x7f00f80b4e70 2024-03-25T18:25:41.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.651+0000 7f00f659c640 10 monclient: get_auth_request con 0x7f00d806e0b0 auth_method 0 2024-03-25T18:25:41.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.651+0000 7f00eeffd640 10 client.4204.objecter ms_handle_connect 0x7f00d806e0b0 2024-03-25T18:25:41.669 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.668+0000 7f00eeffd640 1 -- 172.21.15.3:0/3556230185 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f00f0053910 con 0x7f00f80b4e70 2024-03-25T18:25:41.669 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.668+0000 7f00eeffd640 10 client.4204.objecter ms_dispatch 0x7f00f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:41.669 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.668+0000 7f00eeffd640 3 client.4204.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:41.669 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.668+0000 7f00eeffd640 3 client.4204.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:41.670 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.668+0000 7f00eeffd640 20 client.4204.objecter dump_active .. 0 homeless 2024-03-25T18:25:41.670 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.669+0000 7f00eeffd640 1 -- 172.21.15.3:0/3556230185 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f00f004f250 con 0x7f00f80b4e70 2024-03-25T18:25:41.670 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.669+0000 7f00eeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:41.670 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:41.669+0000 7f00eeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:42.059 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:42.057+0000 7f00fc88f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:42.059 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:42.057+0000 7f00fc88f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:42.059 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:42.057+0000 7f00fc88f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:42.059 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:42.057+0000 7f00fc88f640 1 -- 172.21.15.3:0/3556230185 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00f81423c0 con 0x7f00f80b4e70 2024-03-25T18:25:42.073 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:42.072+0000 7f00eeffd640 1 -- 172.21.15.3:0/3556230185 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f00f0051350 con 0x7f00f80b4e70 2024-03-25T18:25:42.073 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:42.072+0000 7f00eeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:42.073 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:42.072+0000 7f00eeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:42.074 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:25:42.074 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:42.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:42.075+0000 7f00fc88f640 10 librados: watch_flush enter 2024-03-25T18:25:42.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:42.075+0000 7f00fc88f640 10 librados: watch_flush exit 2024-03-25T18:25:42.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:42.075+0000 7f00fc88f640 20 client.4204.objecter shutdown clearing up homeless session... 2024-03-25T18:25:42.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:42.075+0000 7f00fc88f640 10 client.4204.objecter successfully canceled tick 2024-03-25T18:25:42.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:42.075+0000 7f00fc88f640 1 -- 172.21.15.3:0/3556230185 >> v1:172.21.15.3:6800/4195140966 conn(0x7f00d806e0b0 legacy=0x7f00d8070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:42.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:42.075+0000 7f00fc88f640 10 monclient: shutdown 2024-03-25T18:25:42.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:42.075+0000 7f00fc88f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:42.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:42.075+0000 7f00fc88f640 1 -- 172.21.15.3:0/3556230185 >> v1:172.21.15.16:6789/0 conn(0x7f00f80b4e70 legacy=0x7f00f81e6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:42.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:42.075+0000 7f00fc88f640 1 -- 172.21.15.3:0/3556230185 shutdown_connections 2024-03-25T18:25:42.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:42.075+0000 7f00fc88f640 1 -- 172.21.15.3:0/3556230185 >> 172.21.15.3:0/3556230185 conn(0x7f00f80617a0 msgr2=0x7f00f80b6740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:42.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:42.075+0000 7f00fc88f640 1 -- 172.21.15.3:0/3556230185 shutdown_connections 2024-03-25T18:25:42.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:42.076+0000 7f00fc88f640 1 -- 172.21.15.3:0/3556230185 wait complete. 2024-03-25T18:25:42.078 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:42.076+0000 7f00fc88f640 1 librados: shutdown 2024-03-25T18:25:42.092 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:45.148 INFO:tasks.ceph.osd.7.smithi016.stderr:2024-03-25T18:25:45.144+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T18:25:45.188 INFO:tasks.ceph.osd.5.smithi016.stderr:2024-03-25T18:25:45.176+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T18:25:45.264 INFO:tasks.ceph.osd.1.smithi003.stderr:2024-03-25T18:25:45.255+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T18:25:45.353 INFO:tasks.ceph.osd.4.smithi016.stderr:2024-03-25T18:25:45.348+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T18:25:45.470 INFO:tasks.ceph.osd.6.smithi016.stderr:2024-03-25T18:25:45.466+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T18:25:45.778 INFO:tasks.ceph.osd.2.smithi003.stderr:2024-03-25T18:25:45.774+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T18:25:46.178 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-03-25T18:25:46.174+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T18:25:46.473 INFO:tasks.ceph.osd.3.smithi003.stderr:2024-03-25T18:25:46.469+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T18:25:48.898 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:49.014 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.012+0000 7fb2a11e4640 10 monclient: get_monmap_and_config 2024-03-25T18:25:49.014 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.012+0000 7fb2a11e4640 10 monclient: build_initial_monmap 2024-03-25T18:25:49.014 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.012+0000 7fb2a11e4640 10 monclient: monmap: 2024-03-25T18:25:49.014 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:25:49.014 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:49.014 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:25:49.013725+0000 2024-03-25T18:25:49.015 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:25:49.013725+0000 2024-03-25T18:25:49.015 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:49.015 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:49.015 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:49.015 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:25:49.015 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:25:49.015 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:49.017 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.013+0000 7fb2a11e4640 1 Processor -- start 2024-03-25T18:25:49.017 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.013+0000 7fb2a11e4640 1 -- start start 2024-03-25T18:25:49.017 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.013+0000 7fb2a11e4640 10 monclient: init 2024-03-25T18:25:49.017 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.014+0000 7fb2a11e4640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:49.017 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.014+0000 7fb2a11e4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:25:49.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.014+0000 7fb2a11e4640 10 monclient(hunting): picked mon.noname-c con 0x7fb29c06ea30 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:49.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.014+0000 7fb2a11e4640 10 monclient(hunting): picked mon.noname-a con 0x7fb29c06ee10 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:49.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.014+0000 7fb2a11e4640 10 monclient(hunting): picked mon.noname-b con 0x7fb29c06f8e0 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:49.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.014+0000 7fb2a11e4640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb29c13e6d0 con 0x7fb29c06ee10 2024-03-25T18:25:49.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.014+0000 7fb2a11e4640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb29c13c0e0 con 0x7fb29c06ea30 2024-03-25T18:25:49.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.014+0000 7fb2a11e4640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb29c06fe00 con 0x7fb29c06f8e0 2024-03-25T18:25:49.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.014+0000 7fb2a11e4640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:49.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.014+0000 7fb2a11e4640 10 monclient(hunting): authenticate will time out at 7403.274958s 2024-03-25T18:25:49.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.035+0000 7fb293fff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fb29c06ee10 0x7fb29c071910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52646/0 (socket says 172.21.15.3:52646) 2024-03-25T18:25:49.037 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.036+0000 7fb293fff640 1 -- 172.21.15.3:0/1279647357 learned_addr learned my addr 172.21.15.3:0/1279647357 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:25:49.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.046+0000 7fb29a59c640 1 -- 172.21.15.3:0/1279647357 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765661836 0 0) 0x7fb29c13e6d0 con 0x7fb29c06ee10 2024-03-25T18:25:49.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.046+0000 7fb29a59c640 10 monclient(hunting): _negotiate 2024-03-25T18:25:49.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.046+0000 7fb29a59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:49.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.046+0000 7fb29a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:49.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.046+0000 7fb29a59c640 10 monclient(hunting): my global_id is 4266 2024-03-25T18:25:49.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.046+0000 7fb29a59c640 1 -- 172.21.15.3:0/1279647357 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb288003520 con 0x7fb29c06ee10 2024-03-25T18:25:49.054 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.053+0000 7fb29a59c640 1 -- 172.21.15.3:0/1279647357 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1809228574 0 0) 0x7fb28c001110 con 0x7fb29c06ea30 2024-03-25T18:25:49.054 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.053+0000 7fb29a59c640 10 monclient(hunting): _negotiate 2024-03-25T18:25:49.054 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.053+0000 7fb29a59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:49.054 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.053+0000 7fb29a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:49.054 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.053+0000 7fb29a59c640 10 monclient(hunting): my global_id is 4231 2024-03-25T18:25:49.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.053+0000 7fb29a59c640 1 -- 172.21.15.3:0/1279647357 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb29c13e6d0 con 0x7fb29c06ea30 2024-03-25T18:25:49.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.063+0000 7fb29a59c640 1 -- 172.21.15.3:0/1279647357 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2810248541 0 0) 0x7fb29c13e6d0 con 0x7fb29c06ea30 2024-03-25T18:25:49.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.063+0000 7fb29a59c640 1 -- 172.21.15.3:0/1279647357 >> v1:172.21.15.3:6790/0 conn(0x7fb29c06f8e0 legacy=0x7fb29c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:49.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.063+0000 7fb29a59c640 1 -- 172.21.15.3:0/1279647357 >> v1:172.21.15.3:6789/0 conn(0x7fb29c06ee10 legacy=0x7fb29c071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:49.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.063+0000 7fb29a59c640 10 monclient: _finish_hunting 0 2024-03-25T18:25:49.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.063+0000 7fb29a59c640 1 monclient: found mon.noname-c 2024-03-25T18:25:49.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.063+0000 7fb29a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:49.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.063+0000 7fb29a59c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:25:49.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.063+0000 7fb29a59c640 1 -- 172.21.15.3:0/1279647357 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb29c070c80 con 0x7fb29c06ea30 2024-03-25T18:25:49.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.063+0000 7fb29a59c640 10 monclient: _finish_auth 0 2024-03-25T18:25:49.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.063+0000 7fb29a59c640 10 monclient: _check_auth_tickets 2024-03-25T18:25:49.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.063+0000 7fb29a59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:49.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.063+0000 7fb29a59c640 1 -- 172.21.15.3:0/1279647357 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fb28c001660 con 0x7fb29c06ea30 2024-03-25T18:25:49.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.063+0000 7fb2a11e4640 5 monclient: authenticate success, global_id 4231 2024-03-25T18:25:49.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.063+0000 7fb29a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:49.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.063+0000 7fb29a59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:49.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.063+0000 7fb29a59c640 10 monclient: dump: 2024-03-25T18:25:49.065 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:49.065 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:49.065 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:49.066 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:49.066 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:49.066 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:49.066 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:49.066 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:49.066 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:49.066 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:49.066 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.063+0000 7fb2a11e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:49.074 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.070+0000 7fb29a59c640 1 -- 172.21.15.3:0/1279647357 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb28c0017c0 con 0x7fb29c06ea30 2024-03-25T18:25:49.075 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.071+0000 7fb29a59c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:49.075 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.071+0000 7fb29a59c640 1 -- 172.21.15.3:0/1279647357 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fb28c001b50 con 0x7fb29c06ea30 2024-03-25T18:25:49.075 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.071+0000 7fb2a11e4640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:49.075 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.071+0000 7fb2a11e4640 4 set_mon_vals no callback set 2024-03-25T18:25:49.075 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.071+0000 7fb29a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:49.075 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.071+0000 7fb29a59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:49.075 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.071+0000 7fb29a59c640 10 monclient: dump: 2024-03-25T18:25:49.075 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:49.075 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:49.075 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:49.075 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:49.075 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:49.075 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:49.075 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:49.075 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:49.076 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:49.076 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:49.076 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.071+0000 7fb2a11e4640 10 monclient: shutdown 2024-03-25T18:25:49.076 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.071+0000 7fb2a11e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:49.076 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.071+0000 7fb2a11e4640 1 -- 172.21.15.3:0/1279647357 >> v1:172.21.15.16:6789/0 conn(0x7fb29c06ea30 legacy=0x7fb29c13f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:49.076 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.071+0000 7fb2a11e4640 1 -- 172.21.15.3:0/1279647357 shutdown_connections 2024-03-25T18:25:49.076 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.071+0000 7fb2a11e4640 1 -- 172.21.15.3:0/1279647357 >> 172.21.15.3:0/1279647357 conn(0x7fb29c0b3cd0 msgr2=0x7fb29c0b40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:49.076 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.072+0000 7fb2a11e4640 1 -- 172.21.15.3:0/1279647357 shutdown_connections 2024-03-25T18:25:49.076 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.072+0000 7fb2a11e4640 1 -- 172.21.15.3:0/1279647357 wait complete. 2024-03-25T18:25:49.076 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.073+0000 7fb2a11e4640 10 monclient: build_initial_monmap 2024-03-25T18:25:49.076 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.073+0000 7fb2a11e4640 10 monclient: monmap: 2024-03-25T18:25:49.076 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:25:49.076 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:49.076 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:25:49.076 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:25:49.076 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:49.077 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:25:49.077 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:49.077 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:25:49.077 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:25:49.077 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:49.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.073+0000 7fb2a11e4640 1 librados: starting msgr at 2024-03-25T18:25:49.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.073+0000 7fb2a11e4640 1 librados: starting objecter 2024-03-25T18:25:49.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.073+0000 7fb2a11e4640 1 Processor -- start 2024-03-25T18:25:49.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.073+0000 7fb2a11e4640 1 -- start start 2024-03-25T18:25:49.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.073+0000 7fb2a11e4640 1 librados: setting wanted keys 2024-03-25T18:25:49.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.073+0000 7fb2a11e4640 1 librados: calling monclient init 2024-03-25T18:25:49.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.073+0000 7fb2a11e4640 10 monclient: init 2024-03-25T18:25:49.079 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.074+0000 7fb2a11e4640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:49.079 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.074+0000 7fb2a11e4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:49.079 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.074+0000 7fb2a11e4640 10 monclient(hunting): picked mon.noname-a con 0x7fb29c06ea30 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:49.079 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.074+0000 7fb2a11e4640 10 monclient(hunting): picked mon.noname-b con 0x7fb29c0adad0 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:49.079 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.074+0000 7fb2a11e4640 10 monclient(hunting): picked mon.noname-c con 0x7fb29c1daa20 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:49.079 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.074+0000 7fb2a11e4640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb29c134630 con 0x7fb29c06ea30 2024-03-25T18:25:49.079 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.074+0000 7fb2a11e4640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb29c14d540 con 0x7fb29c0adad0 2024-03-25T18:25:49.079 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.074+0000 7fb2a11e4640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb29c1348a0 con 0x7fb29c1daa20 2024-03-25T18:25:49.079 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.074+0000 7fb2a11e4640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:49.079 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.074+0000 7fb2a11e4640 10 monclient(hunting): authenticate will time out at 7403.334947s 2024-03-25T18:25:49.088 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.087+0000 7fb29ad9d640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fb29c06ea30 0x7fb29c14c720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52706/0 (socket says 172.21.15.3:52706) 2024-03-25T18:25:49.088 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.087+0000 7fb29ad9d640 1 -- 172.21.15.3:0/697287467 learned_addr learned my addr 172.21.15.3:0/697287467 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:25:49.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.093+0000 7fb2937fe640 10 client.?.objecter ms_handle_connect 0x7fb29c1daa20 2024-03-25T18:25:49.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.093+0000 7fb2937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:49.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.093+0000 7fb2937fe640 10 client.?.objecter ms_handle_connect 0x7fb29c0adad0 2024-03-25T18:25:49.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.093+0000 7fb2937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:49.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.094+0000 7fb2937fe640 10 client.?.objecter ms_handle_connect 0x7fb29c06ea30 2024-03-25T18:25:49.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.094+0000 7fb2937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:49.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.105+0000 7fb2937fe640 1 -- 172.21.15.3:0/697287467 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135862896 0 0) 0x7fb29c1348a0 con 0x7fb29c1daa20 2024-03-25T18:25:49.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.105+0000 7fb2937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:49.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.105+0000 7fb2937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:49.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.105+0000 7fb2937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:49.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.105+0000 7fb2937fe640 10 monclient(hunting): my global_id is 4241 2024-03-25T18:25:49.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.105+0000 7fb2937fe640 1 -- 172.21.15.3:0/697287467 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb27c004eb0 con 0x7fb29c1daa20 2024-03-25T18:25:49.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.109+0000 7fb2937fe640 1 -- 172.21.15.3:0/697287467 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1007350027 0 0) 0x7fb29c14d540 con 0x7fb29c0adad0 2024-03-25T18:25:49.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.109+0000 7fb2937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:49.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.109+0000 7fb2937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:49.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.109+0000 7fb2937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:49.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.109+0000 7fb2937fe640 10 monclient(hunting): my global_id is 4234 2024-03-25T18:25:49.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.109+0000 7fb2937fe640 1 -- 172.21.15.3:0/697287467 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb29c1348a0 con 0x7fb29c0adad0 2024-03-25T18:25:49.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.121+0000 7fb2937fe640 1 -- 172.21.15.3:0/697287467 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 556102851 0 0) 0x7fb29c134630 con 0x7fb29c06ea30 2024-03-25T18:25:49.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.121+0000 7fb2937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:49.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.121+0000 7fb2937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:49.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.121+0000 7fb2937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:49.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.121+0000 7fb2937fe640 10 monclient(hunting): my global_id is 4269 2024-03-25T18:25:49.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.121+0000 7fb2937fe640 1 -- 172.21.15.3:0/697287467 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb29c14d540 con 0x7fb29c06ea30 2024-03-25T18:25:49.127 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.126+0000 7fb2937fe640 1 -- 172.21.15.3:0/697287467 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 684837250 0 0) 0x7fb27c004eb0 con 0x7fb29c1daa20 2024-03-25T18:25:49.128 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.126+0000 7fb2937fe640 1 -- 172.21.15.3:0/697287467 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb29c134630 con 0x7fb29c1daa20 2024-03-25T18:25:49.128 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.126+0000 7fb2937fe640 1 -- 172.21.15.3:0/697287467 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fb294001b90 con 0x7fb29c1daa20 2024-03-25T18:25:49.128 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.126+0000 7fb2937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:49.128 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.126+0000 7fb2937fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:49.128 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.126+0000 7fb2937fe640 10 monclient(hunting): dump: 2024-03-25T18:25:49.128 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:49.128 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:49.128 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:49.128 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:49.128 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:49.128 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:49.128 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:49.128 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:49.128 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:49.128 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:49.129 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.126+0000 7fb2937fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:49.132 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.130+0000 7fb2937fe640 1 -- 172.21.15.3:0/697287467 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1306993376 0 0) 0x7fb29c1348a0 con 0x7fb29c0adad0 2024-03-25T18:25:49.132 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.130+0000 7fb2937fe640 1 -- 172.21.15.3:0/697287467 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb27c004eb0 con 0x7fb29c0adad0 2024-03-25T18:25:49.132 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.130+0000 7fb2937fe640 1 -- 172.21.15.3:0/697287467 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fb284001950 con 0x7fb29c0adad0 2024-03-25T18:25:49.132 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.130+0000 7fb2937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:49.132 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.130+0000 7fb2937fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:49.132 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.130+0000 7fb2937fe640 10 monclient(hunting): dump: 2024-03-25T18:25:49.132 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:49.132 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:49.132 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:49.132 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:49.133 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:49.133 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:49.133 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:49.133 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:49.133 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:49.133 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:49.133 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.130+0000 7fb2937fe640 1 -- 172.21.15.3:0/697287467 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1358773064 0 0) 0x7fb29c14d540 con 0x7fb29c06ea30 2024-03-25T18:25:49.133 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.130+0000 7fb2937fe640 1 -- 172.21.15.3:0/697287467 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb29c1348a0 con 0x7fb29c06ea30 2024-03-25T18:25:49.133 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.130+0000 7fb2937fe640 1 -- 172.21.15.3:0/697287467 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fb28c0035f0 con 0x7fb29c06ea30 2024-03-25T18:25:49.133 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.130+0000 7fb2937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:49.133 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.130+0000 7fb2937fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:49.133 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.130+0000 7fb2937fe640 10 monclient(hunting): dump: 2024-03-25T18:25:49.133 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:49.133 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:49.134 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:49.134 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:49.134 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:49.134 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:49.134 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:49.134 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:49.134 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:49.134 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:49.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.137+0000 7fb2937fe640 1 -- 172.21.15.3:0/697287467 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1980271879 0 0) 0x7fb27c004eb0 con 0x7fb29c0adad0 2024-03-25T18:25:49.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.137+0000 7fb2937fe640 1 -- 172.21.15.3:0/697287467 >> v1:172.21.15.3:6790/0 conn(0x7fb29c1daa20 legacy=0x7fb29c14ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:49.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.137+0000 7fb2937fe640 1 -- 172.21.15.3:0/697287467 >> v1:172.21.15.3:6789/0 conn(0x7fb29c06ea30 legacy=0x7fb29c14c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:49.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.137+0000 7fb2937fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:49.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.137+0000 7fb2937fe640 1 monclient: found mon.b 2024-03-25T18:25:49.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.137+0000 7fb2937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:49.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.137+0000 7fb2937fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:49.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.137+0000 7fb2937fe640 1 -- 172.21.15.3:0/697287467 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb29c134b80 con 0x7fb29c0adad0 2024-03-25T18:25:49.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.137+0000 7fb2937fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:49.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.137+0000 7fb2937fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:49.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.137+0000 7fb2937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:49.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.137+0000 7fb2a11e4640 5 monclient: authenticate success, global_id 4234 2024-03-25T18:25:49.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.137+0000 7fb2a11e4640 10 monclient: _renew_subs 2024-03-25T18:25:49.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.137+0000 7fb2a11e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:49.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.137+0000 7fb2a11e4640 1 -- 172.21.15.3:0/697287467 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb29c1db560 con 0x7fb29c0adad0 2024-03-25T18:25:49.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.137+0000 7fb2a11e4640 10 client.4234.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:49.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.137+0000 7fb2a11e4640 10 monclient: _renew_subs 2024-03-25T18:25:49.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.137+0000 7fb2a11e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:49.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.137+0000 7fb2a11e4640 1 -- 172.21.15.3:0/697287467 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb29c1e3d70 con 0x7fb29c0adad0 2024-03-25T18:25:49.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.137+0000 7fb2a11e4640 1 librados: init done 2024-03-25T18:25:49.141 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.138+0000 7fb2a11e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:49.141 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.138+0000 7fb2a11e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:49.141 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.138+0000 7fb2a11e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:49.141 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.138+0000 7fb2a11e4640 1 -- 172.21.15.3:0/697287467 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb29c1dae30 con 0x7fb29c0adad0 2024-03-25T18:25:49.359 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.357+0000 7fb2937fe640 1 -- 172.21.15.3:0/697287467 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb284001d60 con 0x7fb29c0adad0 2024-03-25T18:25:49.359 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.357+0000 7fb2937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:49.359 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.357+0000 7fb299d9b640 4 set_mon_vals no callback set 2024-03-25T18:25:49.433 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.432+0000 7fb2937fe640 1 -- 172.21.15.3:0/697287467 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fb284004980 con 0x7fb29c0adad0 2024-03-25T18:25:49.433 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.432+0000 7fb2937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:49.434 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.432+0000 7fb2937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:49.434 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.432+0000 7fb2937fe640 10 monclient: dump: 2024-03-25T18:25:49.434 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:49.434 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:49.434 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:49.434 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:49.434 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:49.434 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:49.434 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:49.434 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:49.434 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:49.434 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:49.435 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.433+0000 7fb2937fe640 1 -- 172.21.15.3:0/697287467 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7fb28401a950 con 0x7fb29c0adad0 2024-03-25T18:25:49.435 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.433+0000 7fb29ad9d640 10 monclient: get_auth_request con 0x7fb27c06e110 auth_method 0 2024-03-25T18:25:49.435 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.434+0000 7fb2937fe640 10 client.4234.objecter ms_handle_connect 0x7fb27c06e110 2024-03-25T18:25:49.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.436+0000 7fb2937fe640 1 -- 172.21.15.3:0/697287467 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7fb284054720 con 0x7fb29c0adad0 2024-03-25T18:25:49.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.437+0000 7fb2937fe640 10 client.4234.objecter ms_dispatch 0x7fb29c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:49.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.437+0000 7fb2937fe640 3 client.4234.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:49.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.437+0000 7fb2937fe640 3 client.4234.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:49.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.437+0000 7fb2937fe640 20 client.4234.objecter dump_active .. 0 homeless 2024-03-25T18:25:49.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.439+0000 7fb2937fe640 1 -- 172.21.15.3:0/697287467 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb284050060 con 0x7fb29c0adad0 2024-03-25T18:25:49.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.439+0000 7fb2937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:49.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.439+0000 7fb2937fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:49.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.693+0000 7fb2a11e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:49.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.693+0000 7fb2a11e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:49.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.693+0000 7fb2a11e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:25:49.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.693+0000 7fb2a11e4640 1 -- 172.21.15.3:0/697287467 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb29c04f730 con 0x7fb29c0adad0 2024-03-25T18:25:49.717 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.716+0000 7fb2937fe640 1 -- 172.21.15.3:0/697287467 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7fb284052160 con 0x7fb29c0adad0 2024-03-25T18:25:49.717 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.716+0000 7fb2937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:49.717 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.716+0000 7fb2937fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:49.718 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:25:49.718 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:49.720 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.718+0000 7fb2a11e4640 10 librados: watch_flush enter 2024-03-25T18:25:49.720 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.718+0000 7fb2a11e4640 10 librados: watch_flush exit 2024-03-25T18:25:49.720 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.718+0000 7fb2a11e4640 20 client.4234.objecter shutdown clearing up homeless session... 2024-03-25T18:25:49.720 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.718+0000 7fb2a11e4640 10 client.4234.objecter successfully canceled tick 2024-03-25T18:25:49.720 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.718+0000 7fb2a11e4640 1 -- 172.21.15.3:0/697287467 >> v1:172.21.15.3:6800/4195140966 conn(0x7fb27c06e110 legacy=0x7fb27c0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:49.720 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.718+0000 7fb2a11e4640 10 monclient: shutdown 2024-03-25T18:25:49.720 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.718+0000 7fb2a11e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:49.720 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.718+0000 7fb2a11e4640 1 -- 172.21.15.3:0/697287467 >> v1:172.21.15.16:6789/0 conn(0x7fb29c0adad0 legacy=0x7fb29c142220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:49.720 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.718+0000 7fb2a11e4640 1 -- 172.21.15.3:0/697287467 shutdown_connections 2024-03-25T18:25:49.721 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.718+0000 7fb2a11e4640 1 -- 172.21.15.3:0/697287467 >> 172.21.15.3:0/697287467 conn(0x7fb29c0b3cd0 msgr2=0x7fb29c0b2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:49.721 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.719+0000 7fb2a11e4640 1 -- 172.21.15.3:0/697287467 shutdown_connections 2024-03-25T18:25:49.721 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.719+0000 7fb2a11e4640 1 -- 172.21.15.3:0/697287467 wait complete. 2024-03-25T18:25:49.721 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:49.719+0000 7fb2a11e4640 1 librados: shutdown 2024-03-25T18:25:49.732 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:56.538 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:56.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.654+0000 7fd5cac2a640 10 monclient: get_monmap_and_config 2024-03-25T18:25:56.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.654+0000 7fd5cac2a640 10 monclient: build_initial_monmap 2024-03-25T18:25:56.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.654+0000 7fd5cac2a640 10 monclient: monmap: 2024-03-25T18:25:56.655 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:25:56.656 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:56.656 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:25:56.655543+0000 2024-03-25T18:25:56.656 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:25:56.655543+0000 2024-03-25T18:25:56.656 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:56.656 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:56.656 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:56.656 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:25:56.656 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:25:56.656 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:56.657 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.655+0000 7fd5cac2a640 1 Processor -- start 2024-03-25T18:25:56.657 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.655+0000 7fd5cac2a640 1 -- start start 2024-03-25T18:25:56.657 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.655+0000 7fd5cac2a640 10 monclient: init 2024-03-25T18:25:56.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.656+0000 7fd5cac2a640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:56.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.656+0000 7fd5cac2a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:56.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.656+0000 7fd5cac2a640 10 monclient(hunting): picked mon.noname-b con 0x7fd5c40b4e70 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:56.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.656+0000 7fd5cac2a640 10 monclient(hunting): picked mon.noname-c con 0x7fd5c40b44b0 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:56.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.656+0000 7fd5cac2a640 10 monclient(hunting): picked mon.noname-a con 0x7fd5c40b1b00 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:56.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.656+0000 7fd5cac2a640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5c40a96f0 con 0x7fd5c40b1b00 2024-03-25T18:25:56.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.656+0000 7fd5cac2a640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5c40ab0e0 con 0x7fd5c40b44b0 2024-03-25T18:25:56.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.656+0000 7fd5cac2a640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5c4145f70 con 0x7fd5c40b4e70 2024-03-25T18:25:56.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.656+0000 7fd5cac2a640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:56.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.656+0000 7fd5cac2a640 10 monclient(hunting): authenticate will time out at 7410.916912s 2024-03-25T18:25:56.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.658+0000 7fd5c91c7640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd5c40b1b00 0x7fd5c414dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57172/0 (socket says 172.21.15.3:57172) 2024-03-25T18:25:56.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.658+0000 7fd5c91c7640 1 -- 172.21.15.3:0/237348537 learned_addr learned my addr 172.21.15.3:0/237348537 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:25:56.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.678+0000 7fd5c37fe640 1 -- 172.21.15.3:0/237348537 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1201825639 0 0) 0x7fd5c40ab0e0 con 0x7fd5c40b44b0 2024-03-25T18:25:56.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.678+0000 7fd5c37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:56.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.678+0000 7fd5c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:56.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.678+0000 7fd5c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:56.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.678+0000 7fd5c37fe640 10 monclient(hunting): my global_id is 4237 2024-03-25T18:25:56.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.678+0000 7fd5c37fe640 1 -- 172.21.15.3:0/237348537 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5b4003520 con 0x7fd5c40b44b0 2024-03-25T18:25:56.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.678+0000 7fd5c37fe640 1 -- 172.21.15.3:0/237348537 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2225625499 0 0) 0x7fd5c40a96f0 con 0x7fd5c40b1b00 2024-03-25T18:25:56.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.678+0000 7fd5c37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:56.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.678+0000 7fd5c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:56.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.678+0000 7fd5c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:56.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.678+0000 7fd5c37fe640 10 monclient(hunting): my global_id is 4272 2024-03-25T18:25:56.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.678+0000 7fd5c37fe640 1 -- 172.21.15.3:0/237348537 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5c40ab0e0 con 0x7fd5c40b1b00 2024-03-25T18:25:56.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.678+0000 7fd5c37fe640 1 -- 172.21.15.3:0/237348537 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 111561911 0 0) 0x7fd5c4145f70 con 0x7fd5c40b4e70 2024-03-25T18:25:56.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.678+0000 7fd5c37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:56.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.678+0000 7fd5c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:56.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.678+0000 7fd5c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:56.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.678+0000 7fd5c37fe640 10 monclient(hunting): my global_id is 4244 2024-03-25T18:25:56.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.678+0000 7fd5c37fe640 1 -- 172.21.15.3:0/237348537 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5c40a96f0 con 0x7fd5c40b4e70 2024-03-25T18:25:56.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.685+0000 7fd5c37fe640 1 -- 172.21.15.3:0/237348537 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2520091286 0 0) 0x7fd5c40a96f0 con 0x7fd5c40b4e70 2024-03-25T18:25:56.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.686+0000 7fd5c37fe640 1 -- 172.21.15.3:0/237348537 >> v1:172.21.15.16:6789/0 conn(0x7fd5c40b44b0 legacy=0x7fd5c4145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:56.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.686+0000 7fd5c37fe640 1 -- 172.21.15.3:0/237348537 >> v1:172.21.15.3:6789/0 conn(0x7fd5c40b1b00 legacy=0x7fd5c414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:56.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.686+0000 7fd5c37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:56.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.686+0000 7fd5c37fe640 1 monclient: found mon.noname-b 2024-03-25T18:25:56.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.686+0000 7fd5c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:56.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.686+0000 7fd5c37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-25T18:25:56.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.686+0000 7fd5c37fe640 1 -- 172.21.15.3:0/237348537 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c40a9b80 con 0x7fd5c40b4e70 2024-03-25T18:25:56.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.686+0000 7fd5c37fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:56.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.686+0000 7fd5c37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:56.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.686+0000 7fd5c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:56.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.686+0000 7fd5c37fe640 1 -- 172.21.15.3:0/237348537 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd5b80015b0 con 0x7fd5c40b4e70 2024-03-25T18:25:56.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.686+0000 7fd5cac2a640 5 monclient: authenticate success, global_id 4244 2024-03-25T18:25:56.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.686+0000 7fd5cac2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:56.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.686+0000 7fd5c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:56.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.686+0000 7fd5c37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:56.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.686+0000 7fd5c37fe640 10 monclient: dump: 2024-03-25T18:25:56.688 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:56.688 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:56.688 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:56.688 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:56.688 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:56.688 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:56.689 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:56.689 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:56.689 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:56.689 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:56.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.686+0000 7fd5cac2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:56.721 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.720+0000 7fd5c37fe640 1 -- 172.21.15.3:0/237348537 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5b8001710 con 0x7fd5c40b4e70 2024-03-25T18:25:56.721 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.720+0000 7fd5c37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:56.721 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.720+0000 7fd5c37fe640 1 -- 172.21.15.3:0/237348537 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd5b8001b60 con 0x7fd5c40b4e70 2024-03-25T18:25:56.721 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.720+0000 7fd5c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:56.721 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.720+0000 7fd5c37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:56.721 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.720+0000 7fd5c37fe640 10 monclient: dump: 2024-03-25T18:25:56.721 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:56.722 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:56.722 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:56.722 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:56.722 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:56.722 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:56.722 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:56.722 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:56.722 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:56.722 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:56.722 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.720+0000 7fd5cac2a640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:56.722 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.720+0000 7fd5cac2a640 4 set_mon_vals no callback set 2024-03-25T18:25:56.724 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.720+0000 7fd5cac2a640 10 monclient: shutdown 2024-03-25T18:25:56.724 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.720+0000 7fd5cac2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:56.724 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.720+0000 7fd5cac2a640 1 -- 172.21.15.3:0/237348537 >> v1:172.21.15.3:6790/0 conn(0x7fd5c40b4e70 legacy=0x7fd5c413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:56.724 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.720+0000 7fd5cac2a640 1 -- 172.21.15.3:0/237348537 shutdown_connections 2024-03-25T18:25:56.724 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.720+0000 7fd5cac2a640 1 -- 172.21.15.3:0/237348537 >> 172.21.15.3:0/237348537 conn(0x7fd5c40617a0 msgr2=0x7fd5c4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:56.724 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.721+0000 7fd5cac2a640 1 -- 172.21.15.3:0/237348537 shutdown_connections 2024-03-25T18:25:56.725 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.721+0000 7fd5cac2a640 1 -- 172.21.15.3:0/237348537 wait complete. 2024-03-25T18:25:56.725 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.721+0000 7fd5cac2a640 10 monclient: build_initial_monmap 2024-03-25T18:25:56.725 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.721+0000 7fd5cac2a640 10 monclient: monmap: 2024-03-25T18:25:56.725 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:25:56.725 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:56.725 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:25:56.725 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:25:56.725 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:56.725 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:25:56.725 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:25:56.725 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:25:56.725 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:25:56.725 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:56.725 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.721+0000 7fd5cac2a640 1 librados: starting msgr at 2024-03-25T18:25:56.726 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.721+0000 7fd5cac2a640 1 librados: starting objecter 2024-03-25T18:25:56.726 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.721+0000 7fd5cac2a640 1 Processor -- start 2024-03-25T18:25:56.726 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.722+0000 7fd5cac2a640 1 -- start start 2024-03-25T18:25:56.726 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.722+0000 7fd5cac2a640 1 librados: setting wanted keys 2024-03-25T18:25:56.726 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.722+0000 7fd5cac2a640 1 librados: calling monclient init 2024-03-25T18:25:56.726 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.722+0000 7fd5cac2a640 10 monclient: init 2024-03-25T18:25:56.726 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.722+0000 7fd5cac2a640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:56.726 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.722+0000 7fd5cac2a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:25:56.726 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.722+0000 7fd5cac2a640 10 monclient(hunting): picked mon.noname-b con 0x7fd5c40b1b00 addr v1:172.21.15.16:6789/0 2024-03-25T18:25:56.726 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.722+0000 7fd5cac2a640 10 monclient(hunting): picked mon.noname-a con 0x7fd5c40b44b0 addr v1:172.21.15.3:6789/0 2024-03-25T18:25:56.726 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.722+0000 7fd5cac2a640 10 monclient(hunting): picked mon.noname-c con 0x7fd5c40b4e70 addr v1:172.21.15.3:6790/0 2024-03-25T18:25:56.726 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.722+0000 7fd5cac2a640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5c41d7750 con 0x7fd5c40b44b0 2024-03-25T18:25:56.726 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.722+0000 7fd5cac2a640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5c4140400 con 0x7fd5c40b1b00 2024-03-25T18:25:56.727 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.722+0000 7fd5cac2a640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5c4144290 con 0x7fd5c40b4e70 2024-03-25T18:25:56.727 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.722+0000 7fd5cac2a640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:56.727 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.722+0000 7fd5cac2a640 10 monclient(hunting): authenticate will time out at 7410.983077s 2024-03-25T18:25:56.731 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.730+0000 7fd5c89c6640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd5c40b1b00 0x7fd5c4143b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:48448/0 (socket says 172.21.15.3:48448) 2024-03-25T18:25:56.731 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.730+0000 7fd5c89c6640 1 -- 172.21.15.3:0/4186595778 learned_addr learned my addr 172.21.15.3:0/4186595778 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:25:56.734 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.733+0000 7fd5c17fa640 10 client.?.objecter ms_handle_connect 0x7fd5c40b1b00 2024-03-25T18:25:56.734 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.733+0000 7fd5c17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:56.740 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.739+0000 7fd5c17fa640 10 client.?.objecter ms_handle_connect 0x7fd5c40b44b0 2024-03-25T18:25:56.740 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.739+0000 7fd5c17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:56.741 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.740+0000 7fd5c17fa640 1 -- 172.21.15.3:0/4186595778 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1469406040 0 0) 0x7fd5c4140400 con 0x7fd5c40b1b00 2024-03-25T18:25:56.741 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.740+0000 7fd5c17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:56.741 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.740+0000 7fd5c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:56.741 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.740+0000 7fd5c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:56.741 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.740+0000 7fd5c17fa640 10 monclient(hunting): my global_id is 4240 2024-03-25T18:25:56.741 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.740+0000 7fd5c17fa640 1 -- 172.21.15.3:0/4186595778 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5a8004eb0 con 0x7fd5c40b1b00 2024-03-25T18:25:56.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.744+0000 7fd5c17fa640 10 client.?.objecter ms_handle_connect 0x7fd5c40b4e70 2024-03-25T18:25:56.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.745+0000 7fd5c17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:56.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.759+0000 7fd5c17fa640 1 -- 172.21.15.3:0/4186595778 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 770059752 0 0) 0x7fd5a8004eb0 con 0x7fd5c40b1b00 2024-03-25T18:25:56.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.759+0000 7fd5c17fa640 1 -- 172.21.15.3:0/4186595778 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5c4140400 con 0x7fd5c40b1b00 2024-03-25T18:25:56.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.759+0000 7fd5c17fa640 1 -- 172.21.15.3:0/4186595778 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd5b8001410 con 0x7fd5c40b1b00 2024-03-25T18:25:56.761 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.759+0000 7fd5c17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:56.761 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.759+0000 7fd5c17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:56.761 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.759+0000 7fd5c17fa640 10 monclient(hunting): dump: 2024-03-25T18:25:56.761 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:56.761 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:56.761 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:56.761 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:56.761 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:56.761 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:56.761 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:56.761 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:56.761 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:56.761 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:56.761 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.759+0000 7fd5c17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:56.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.760+0000 7fd5c17fa640 1 -- 172.21.15.3:0/4186595778 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231881461 0 0) 0x7fd5c41d7750 con 0x7fd5c40b44b0 2024-03-25T18:25:56.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.760+0000 7fd5c17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:56.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.760+0000 7fd5c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:56.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.760+0000 7fd5c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:56.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.760+0000 7fd5c17fa640 10 monclient(hunting): my global_id is 4275 2024-03-25T18:25:56.763 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.760+0000 7fd5c17fa640 1 -- 172.21.15.3:0/4186595778 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5a8004eb0 con 0x7fd5c40b44b0 2024-03-25T18:25:56.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.765+0000 7fd5c17fa640 1 -- 172.21.15.3:0/4186595778 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1965638668 0 0) 0x7fd5c4144290 con 0x7fd5c40b4e70 2024-03-25T18:25:56.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.765+0000 7fd5c17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:56.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.765+0000 7fd5c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:56.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.765+0000 7fd5c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:56.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.765+0000 7fd5c17fa640 10 monclient(hunting): my global_id is 4247 2024-03-25T18:25:56.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.765+0000 7fd5c17fa640 1 -- 172.21.15.3:0/4186595778 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5c41d7750 con 0x7fd5c40b4e70 2024-03-25T18:25:56.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.775+0000 7fd5c17fa640 1 -- 172.21.15.3:0/4186595778 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 879140026 0 0) 0x7fd5a8004eb0 con 0x7fd5c40b44b0 2024-03-25T18:25:56.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.775+0000 7fd5c17fa640 1 -- 172.21.15.3:0/4186595778 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5c4144290 con 0x7fd5c40b44b0 2024-03-25T18:25:56.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.775+0000 7fd5c17fa640 1 -- 172.21.15.3:0/4186595778 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd5b0001950 con 0x7fd5c40b44b0 2024-03-25T18:25:56.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.775+0000 7fd5c17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:56.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.775+0000 7fd5c17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:56.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.775+0000 7fd5c17fa640 10 monclient(hunting): dump: 2024-03-25T18:25:56.777 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:56.777 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:56.777 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:56.777 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:56.777 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:56.777 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:56.777 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:56.777 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:56.777 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:56.777 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:56.798 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.796+0000 7fd5c17fa640 1 -- 172.21.15.3:0/4186595778 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2947515097 0 0) 0x7fd5c4144290 con 0x7fd5c40b44b0 2024-03-25T18:25:56.798 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.796+0000 7fd5c17fa640 1 -- 172.21.15.3:0/4186595778 >> v1:172.21.15.3:6790/0 conn(0x7fd5c40b4e70 legacy=0x7fd5c41d7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:56.798 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.796+0000 7fd5c17fa640 1 -- 172.21.15.3:0/4186595778 >> v1:172.21.15.16:6789/0 conn(0x7fd5c40b1b00 legacy=0x7fd5c4143b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:56.798 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.796+0000 7fd5c17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:25:56.798 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.796+0000 7fd5c17fa640 1 monclient: found mon.a 2024-03-25T18:25:56.798 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.796+0000 7fd5c17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:56.798 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.796+0000 7fd5c17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:56.798 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.796+0000 7fd5c17fa640 1 -- 172.21.15.3:0/4186595778 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c4144420 con 0x7fd5c40b44b0 2024-03-25T18:25:56.798 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.796+0000 7fd5c17fa640 10 monclient: _finish_auth 0 2024-03-25T18:25:56.798 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.796+0000 7fd5c17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:25:56.798 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.796+0000 7fd5c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:56.798 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.796+0000 7fd5cac2a640 5 monclient: authenticate success, global_id 4275 2024-03-25T18:25:56.799 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.796+0000 7fd5cac2a640 10 monclient: _renew_subs 2024-03-25T18:25:56.799 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.796+0000 7fd5cac2a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:56.799 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.796+0000 7fd5cac2a640 1 -- 172.21.15.3:0/4186595778 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5c4144da0 con 0x7fd5c40b44b0 2024-03-25T18:25:56.799 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.796+0000 7fd5cac2a640 10 client.4275.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:56.799 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.796+0000 7fd5cac2a640 10 monclient: _renew_subs 2024-03-25T18:25:56.799 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.796+0000 7fd5cac2a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:56.799 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.796+0000 7fd5cac2a640 1 -- 172.21.15.3:0/4186595778 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5c41db920 con 0x7fd5c40b44b0 2024-03-25T18:25:56.799 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.796+0000 7fd5cac2a640 1 librados: init done 2024-03-25T18:25:56.799 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.797+0000 7fd5cac2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:56.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.797+0000 7fd5cac2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:56.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.797+0000 7fd5cac2a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:56.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:56.797+0000 7fd5cac2a640 1 -- 172.21.15.3:0/4186595778 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5c4144620 con 0x7fd5c40b44b0 2024-03-25T18:25:57.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.121+0000 7fd5c17fa640 1 -- 172.21.15.3:0/4186595778 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5b0001d60 con 0x7fd5c40b44b0 2024-03-25T18:25:57.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.121+0000 7fd5c17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:57.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.121+0000 7fd5c17fa640 1 -- 172.21.15.3:0/4186595778 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd5b0003790 con 0x7fd5c40b44b0 2024-03-25T18:25:57.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.121+0000 7fd5c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:57.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.121+0000 7fd5c17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:57.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.121+0000 7fd5c17fa640 10 monclient: dump: 2024-03-25T18:25:57.123 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:25:57.123 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:25:57.123 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:57.123 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:25:57.123 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:57.123 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:25:57.123 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:25:57.123 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:25:57.123 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:25:57.123 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:25:57.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.121+0000 7fd5c2ffd640 4 set_mon_vals no callback set 2024-03-25T18:25:57.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.121+0000 7fd5c17fa640 1 -- 172.21.15.3:0/4186595778 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7fd5b0019760 con 0x7fd5c40b44b0 2024-03-25T18:25:57.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.122+0000 7fd5c89c6640 10 monclient: get_auth_request con 0x7fd5a806df30 auth_method 0 2024-03-25T18:25:57.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.122+0000 7fd5c17fa640 10 client.4275.objecter ms_handle_connect 0x7fd5a806df30 2024-03-25T18:25:57.128 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.128+0000 7fd5c17fa640 1 -- 172.21.15.3:0/4186595778 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7fd5b0053530 con 0x7fd5c40b44b0 2024-03-25T18:25:57.128 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.128+0000 7fd5c17fa640 10 client.4275.objecter ms_dispatch 0x7fd5c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:57.128 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.128+0000 7fd5c17fa640 3 client.4275.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:57.128 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.128+0000 7fd5c17fa640 3 client.4275.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:57.128 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.128+0000 7fd5c17fa640 20 client.4275.objecter dump_active .. 0 homeless 2024-03-25T18:25:57.128 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.128+0000 7fd5c17fa640 1 -- 172.21.15.3:0/4186595778 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd5b004ee70 con 0x7fd5c40b44b0 2024-03-25T18:25:57.129 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.128+0000 7fd5c17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:57.129 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.128+0000 7fd5c17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:57.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.381+0000 7fd5cac2a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:57.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.381+0000 7fd5cac2a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:57.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.381+0000 7fd5cac2a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:25:57.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.381+0000 7fd5cac2a640 1 -- 172.21.15.3:0/4186595778 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5c41397f0 con 0x7fd5c40b44b0 2024-03-25T18:25:57.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.404+0000 7fd5c17fa640 1 -- 172.21.15.3:0/4186595778 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7fd5b0050f70 con 0x7fd5c40b44b0 2024-03-25T18:25:57.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.404+0000 7fd5c17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:57.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.404+0000 7fd5c17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:57.404 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:25:57.404 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:57.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.407+0000 7fd5cac2a640 10 librados: watch_flush enter 2024-03-25T18:25:57.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.407+0000 7fd5cac2a640 10 librados: watch_flush exit 2024-03-25T18:25:57.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.407+0000 7fd5cac2a640 20 client.4275.objecter shutdown clearing up homeless session... 2024-03-25T18:25:57.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.407+0000 7fd5cac2a640 10 client.4275.objecter successfully canceled tick 2024-03-25T18:25:57.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.407+0000 7fd5cac2a640 1 -- 172.21.15.3:0/4186595778 >> v1:172.21.15.3:6800/4195140966 conn(0x7fd5a806df30 legacy=0x7fd5a80703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:57.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.407+0000 7fd5cac2a640 10 monclient: shutdown 2024-03-25T18:25:57.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.407+0000 7fd5cac2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:57.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.407+0000 7fd5cac2a640 1 -- 172.21.15.3:0/4186595778 >> v1:172.21.15.3:6789/0 conn(0x7fd5c40b44b0 legacy=0x7fd5c41db030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:57.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.407+0000 7fd5cac2a640 1 -- 172.21.15.3:0/4186595778 shutdown_connections 2024-03-25T18:25:57.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.407+0000 7fd5cac2a640 1 -- 172.21.15.3:0/4186595778 >> 172.21.15.3:0/4186595778 conn(0x7fd5c40617a0 msgr2=0x7fd5c40b6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:57.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.407+0000 7fd5cac2a640 1 -- 172.21.15.3:0/4186595778 shutdown_connections 2024-03-25T18:25:57.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.407+0000 7fd5cac2a640 1 -- 172.21.15.3:0/4186595778 wait complete. 2024-03-25T18:25:57.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:25:57.407+0000 7fd5cac2a640 1 librados: shutdown 2024-03-25T18:25:57.419 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:00.714 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:01:14.443 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:00.792 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:01:14.521 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:00.812 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:01:14.504 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:00.816 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:01:14.545 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:00.882 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:01:14.574 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:00.886 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:01:14.615 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:01.028 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:01:14.757 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:01.109 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:01:14.801 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:01.150 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:01:14.878 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:04.226 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:04.341 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.340+0000 7fbb02086640 10 monclient: get_monmap_and_config 2024-03-25T18:26:04.341 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.340+0000 7fbb02086640 10 monclient: build_initial_monmap 2024-03-25T18:26:04.341 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.340+0000 7fbb02086640 10 monclient: monmap: 2024-03-25T18:26:04.341 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:26:04.341 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:04.342 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:26:04.341467+0000 2024-03-25T18:26:04.342 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:26:04.341467+0000 2024-03-25T18:26:04.342 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:04.342 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:04.342 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:26:04.342 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:26:04.342 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:26:04.342 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:04.343 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.341+0000 7fbb02086640 1 Processor -- start 2024-03-25T18:26:04.343 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.341+0000 7fbb02086640 1 -- start start 2024-03-25T18:26:04.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.341+0000 7fbb02086640 10 monclient: init 2024-03-25T18:26:04.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.342+0000 7fbb02086640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:04.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.342+0000 7fbb02086640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:04.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.342+0000 7fbb02086640 10 monclient(hunting): picked mon.noname-a con 0x7fbafc0b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:26:04.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.342+0000 7fbb02086640 10 monclient(hunting): picked mon.noname-c con 0x7fbafc0b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:26:04.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.342+0000 7fbb02086640 10 monclient(hunting): picked mon.noname-b con 0x7fbafc13c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:26:04.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.342+0000 7fbb02086640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbafc0ab0e0 con 0x7fbafc0b4ff0 2024-03-25T18:26:04.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.342+0000 7fbb02086640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbafc0ab3c0 con 0x7fbafc0b4670 2024-03-25T18:26:04.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.342+0000 7fbb02086640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbafc0a9f30 con 0x7fbafc13c480 2024-03-25T18:26:04.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.342+0000 7fbb02086640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:04.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.342+0000 7fbb02086640 10 monclient(hunting): authenticate will time out at 7418.602625s 2024-03-25T18:26:04.365 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.364+0000 7fbafb7fe640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fbafc0b4ff0 0x7fbafc13bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:49600/0 (socket says 172.21.15.3:49600) 2024-03-25T18:26:04.365 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.364+0000 7fbafb7fe640 1 -- 172.21.15.3:0/3599347964 learned_addr learned my addr 172.21.15.3:0/3599347964 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:26:04.373 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.372+0000 7fbafa7fc640 1 -- 172.21.15.3:0/3599347964 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2615643333 0 0) 0x7fbafc0ab0e0 con 0x7fbafc0b4ff0 2024-03-25T18:26:04.373 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.372+0000 7fbafa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:04.373 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.372+0000 7fbafa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:04.373 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.372+0000 7fbafa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:04.373 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.372+0000 7fbafa7fc640 10 monclient(hunting): my global_id is 4278 2024-03-25T18:26:04.373 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.372+0000 7fbafa7fc640 1 -- 172.21.15.3:0/3599347964 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbae4003520 con 0x7fbafc0b4ff0 2024-03-25T18:26:04.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.377+0000 7fbafa7fc640 1 -- 172.21.15.3:0/3599347964 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 777647031 0 0) 0x7fbafc0a9f30 con 0x7fbafc13c480 2024-03-25T18:26:04.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.377+0000 7fbafa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:04.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.377+0000 7fbafa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:04.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.377+0000 7fbafa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:04.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.377+0000 7fbafa7fc640 10 monclient(hunting): my global_id is 4250 2024-03-25T18:26:04.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.377+0000 7fbafa7fc640 1 -- 172.21.15.3:0/3599347964 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbafc0ab0e0 con 0x7fbafc13c480 2024-03-25T18:26:04.385 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.384+0000 7fbafa7fc640 1 -- 172.21.15.3:0/3599347964 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2716417939 0 0) 0x7fbafc0ab3c0 con 0x7fbafc0b4670 2024-03-25T18:26:04.385 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.384+0000 7fbafa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:04.385 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.384+0000 7fbafa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:04.385 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.384+0000 7fbafa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:04.385 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.384+0000 7fbafa7fc640 10 monclient(hunting): my global_id is 4243 2024-03-25T18:26:04.385 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.384+0000 7fbafa7fc640 1 -- 172.21.15.3:0/3599347964 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbafc0a9f30 con 0x7fbafc0b4670 2024-03-25T18:26:04.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.389+0000 7fbafa7fc640 1 -- 172.21.15.3:0/3599347964 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 802320537 0 0) 0x7fbafc0ab0e0 con 0x7fbafc13c480 2024-03-25T18:26:04.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.389+0000 7fbafa7fc640 1 -- 172.21.15.3:0/3599347964 >> v1:172.21.15.16:6789/0 conn(0x7fbafc0b4670 legacy=0x7fbafc1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:04.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.389+0000 7fbafa7fc640 1 -- 172.21.15.3:0/3599347964 >> v1:172.21.15.3:6789/0 conn(0x7fbafc0b4ff0 legacy=0x7fbafc13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:04.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.389+0000 7fbafa7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:26:04.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.389+0000 7fbafa7fc640 1 monclient: found mon.noname-b 2024-03-25T18:26:04.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.389+0000 7fbafa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:04.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.389+0000 7fbafa7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-25T18:26:04.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.389+0000 7fbafa7fc640 1 -- 172.21.15.3:0/3599347964 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbafc0a9b70 con 0x7fbafc13c480 2024-03-25T18:26:04.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.389+0000 7fbafa7fc640 10 monclient: _finish_auth 0 2024-03-25T18:26:04.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.389+0000 7fbafa7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:26:04.392 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.389+0000 7fbafa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:04.392 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.389+0000 7fbafa7fc640 1 -- 172.21.15.3:0/3599347964 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fbaf40019f0 con 0x7fbafc13c480 2024-03-25T18:26:04.392 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.389+0000 7fbafa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:04.392 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.389+0000 7fbafa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:04.392 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.389+0000 7fbafa7fc640 10 monclient: dump: 2024-03-25T18:26:04.392 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:04.392 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:04.392 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:04.392 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:04.392 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:04.392 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:04.392 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:04.392 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:04.392 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:04.393 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:04.393 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.389+0000 7fbb02086640 5 monclient: authenticate success, global_id 4250 2024-03-25T18:26:04.393 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.389+0000 7fbb02086640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:04.402 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.401+0000 7fbafa7fc640 1 -- 172.21.15.3:0/3599347964 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbaf4001b50 con 0x7fbafc13c480 2024-03-25T18:26:04.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.401+0000 7fbafa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:04.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.401+0000 7fbafa7fc640 1 -- 172.21.15.3:0/3599347964 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fbaf4001f60 con 0x7fbafc13c480 2024-03-25T18:26:04.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.401+0000 7fbafa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:04.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.401+0000 7fbafa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:04.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.401+0000 7fbafa7fc640 10 monclient: dump: 2024-03-25T18:26:04.403 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:04.403 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:04.403 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:04.403 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:04.403 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:04.403 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:04.403 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:04.403 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:04.403 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:04.404 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:04.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.401+0000 7fbb02086640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:04.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.401+0000 7fbb02086640 4 set_mon_vals no callback set 2024-03-25T18:26:04.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.402+0000 7fbb02086640 10 monclient: shutdown 2024-03-25T18:26:04.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.402+0000 7fbb02086640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:04.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.402+0000 7fbb02086640 1 -- 172.21.15.3:0/3599347964 >> v1:172.21.15.3:6790/0 conn(0x7fbafc13c480 legacy=0x7fbafc14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:04.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.402+0000 7fbb02086640 1 -- 172.21.15.3:0/3599347964 shutdown_connections 2024-03-25T18:26:04.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.402+0000 7fbb02086640 1 -- 172.21.15.3:0/3599347964 >> 172.21.15.3:0/3599347964 conn(0x7fbafc0617a0 msgr2=0x7fbafc061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:04.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.403+0000 7fbb02086640 1 -- 172.21.15.3:0/3599347964 shutdown_connections 2024-03-25T18:26:04.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.403+0000 7fbb02086640 1 -- 172.21.15.3:0/3599347964 wait complete. 2024-03-25T18:26:04.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.403+0000 7fbb02086640 10 monclient: build_initial_monmap 2024-03-25T18:26:04.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.403+0000 7fbb02086640 10 monclient: monmap: 2024-03-25T18:26:04.406 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:26:04.406 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:04.406 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:26:04.406 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:26:04.406 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:04.406 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:26:04.406 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:26:04.406 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:26:04.406 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:26:04.406 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:04.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.404+0000 7fbb02086640 1 librados: starting msgr at 2024-03-25T18:26:04.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.404+0000 7fbb02086640 1 librados: starting objecter 2024-03-25T18:26:04.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.404+0000 7fbb02086640 1 Processor -- start 2024-03-25T18:26:04.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.404+0000 7fbb02086640 1 -- start start 2024-03-25T18:26:04.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.404+0000 7fbb02086640 1 librados: setting wanted keys 2024-03-25T18:26:04.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.404+0000 7fbb02086640 1 librados: calling monclient init 2024-03-25T18:26:04.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.404+0000 7fbb02086640 10 monclient: init 2024-03-25T18:26:04.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.405+0000 7fbb02086640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:04.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.405+0000 7fbb02086640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:04.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.405+0000 7fbb02086640 10 monclient(hunting): picked mon.noname-a con 0x7fbafc0b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:26:04.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.405+0000 7fbb02086640 10 monclient(hunting): picked mon.noname-c con 0x7fbafc0b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:26:04.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.405+0000 7fbb02086640 10 monclient(hunting): picked mon.noname-b con 0x7fbafc13c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:26:04.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.405+0000 7fbb02086640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbafc0b0140 con 0x7fbafc0b4670 2024-03-25T18:26:04.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.405+0000 7fbb02086640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbafc0b0420 con 0x7fbafc13c480 2024-03-25T18:26:04.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.405+0000 7fbb02086640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbafc0b0700 con 0x7fbafc0b4ff0 2024-03-25T18:26:04.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.405+0000 7fbb02086640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:04.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.405+0000 7fbb02086640 10 monclient(hunting): authenticate will time out at 7418.665938s 2024-03-25T18:26:04.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.410+0000 7fbafbfff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbafc13c480 0x7fbafc1e2380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:33474/0 (socket says 172.21.15.3:33474) 2024-03-25T18:26:04.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.411+0000 7fbafbfff640 1 -- 172.21.15.3:0/3786960486 learned_addr learned my addr 172.21.15.3:0/3786960486 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:26:04.419 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.418+0000 7fbae3fff640 10 client.?.objecter ms_handle_connect 0x7fbafc13c480 2024-03-25T18:26:04.419 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.418+0000 7fbae3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:04.423 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.422+0000 7fbae3fff640 10 client.?.objecter ms_handle_connect 0x7fbafc0b4670 2024-03-25T18:26:04.423 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.422+0000 7fbae3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:04.426 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.425+0000 7fbae3fff640 1 -- 172.21.15.3:0/3786960486 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340152763 0 0) 0x7fbafc0b0420 con 0x7fbafc13c480 2024-03-25T18:26:04.426 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.425+0000 7fbae3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:04.426 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.425+0000 7fbae3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:04.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.425+0000 7fbae3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:04.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.425+0000 7fbae3fff640 10 monclient(hunting): my global_id is 4246 2024-03-25T18:26:04.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.425+0000 7fbae3fff640 1 -- 172.21.15.3:0/3786960486 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbae8004eb0 con 0x7fbafc13c480 2024-03-25T18:26:04.432 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.431+0000 7fbae3fff640 10 client.?.objecter ms_handle_connect 0x7fbafc0b4ff0 2024-03-25T18:26:04.432 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.431+0000 7fbae3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:04.433 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.432+0000 7fbae3fff640 1 -- 172.21.15.3:0/3786960486 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3259731357 0 0) 0x7fbae8004eb0 con 0x7fbafc13c480 2024-03-25T18:26:04.433 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.432+0000 7fbae3fff640 1 -- 172.21.15.3:0/3786960486 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbafc0b0420 con 0x7fbafc13c480 2024-03-25T18:26:04.434 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.432+0000 7fbae3fff640 1 -- 172.21.15.3:0/3786960486 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fbaf4001fa0 con 0x7fbafc13c480 2024-03-25T18:26:04.434 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.432+0000 7fbae3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:04.434 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.432+0000 7fbae3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:04.434 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.432+0000 7fbae3fff640 10 monclient(hunting): dump: 2024-03-25T18:26:04.434 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:04.434 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:04.434 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:04.434 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:04.434 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:04.434 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:04.434 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:04.434 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:04.434 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:04.435 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:04.435 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.432+0000 7fbae3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:04.437 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.437+0000 7fbae3fff640 1 -- 172.21.15.3:0/3786960486 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1650447759 0 0) 0x7fbafc0b0140 con 0x7fbafc0b4670 2024-03-25T18:26:04.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.437+0000 7fbae3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:04.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.437+0000 7fbae3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:04.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.437+0000 7fbae3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:04.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.437+0000 7fbae3fff640 10 monclient(hunting): my global_id is 4281 2024-03-25T18:26:04.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.437+0000 7fbae3fff640 1 -- 172.21.15.3:0/3786960486 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbae8004eb0 con 0x7fbafc0b4670 2024-03-25T18:26:04.443 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.441+0000 7fbae3fff640 1 -- 172.21.15.3:0/3786960486 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1021546521 0 0) 0x7fbafc0b0420 con 0x7fbafc13c480 2024-03-25T18:26:04.443 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.441+0000 7fbae3fff640 1 -- 172.21.15.3:0/3786960486 >> v1:172.21.15.3:6790/0 conn(0x7fbafc0b4ff0 legacy=0x7fbafc0b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:04.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.441+0000 7fbae3fff640 1 -- 172.21.15.3:0/3786960486 >> v1:172.21.15.3:6789/0 conn(0x7fbafc0b4670 legacy=0x7fbafc0ae310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:04.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.441+0000 7fbae3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:26:04.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.441+0000 7fbae3fff640 1 monclient: found mon.b 2024-03-25T18:26:04.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.441+0000 7fbae3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:04.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.441+0000 7fbae3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:26:04.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.441+0000 7fbae3fff640 1 -- 172.21.15.3:0/3786960486 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbafc0aeb40 con 0x7fbafc13c480 2024-03-25T18:26:04.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.441+0000 7fbae3fff640 10 monclient: _finish_auth 0 2024-03-25T18:26:04.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.441+0000 7fbae3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:26:04.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.441+0000 7fbae3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:04.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.441+0000 7fbb02086640 5 monclient: authenticate success, global_id 4246 2024-03-25T18:26:04.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.441+0000 7fbb02086640 10 monclient: _renew_subs 2024-03-25T18:26:04.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.441+0000 7fbb02086640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:26:04.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.441+0000 7fbb02086640 1 -- 172.21.15.3:0/3786960486 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbafc0aee20 con 0x7fbafc13c480 2024-03-25T18:26:04.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.441+0000 7fbb02086640 10 client.4246.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:04.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.441+0000 7fbb02086640 10 monclient: _renew_subs 2024-03-25T18:26:04.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.441+0000 7fbb02086640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:26:04.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.441+0000 7fbb02086640 1 -- 172.21.15.3:0/3786960486 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbafc0b1480 con 0x7fbafc13c480 2024-03-25T18:26:04.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.442+0000 7fbb02086640 1 librados: init done 2024-03-25T18:26:04.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.442+0000 7fbb02086640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:04.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.443+0000 7fbb02086640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:04.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.443+0000 7fbb02086640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:26:04.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.443+0000 7fbb02086640 1 -- 172.21.15.3:0/3786960486 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbafc06dbd0 con 0x7fbafc13c480 2024-03-25T18:26:04.730 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.729+0000 7fbae3fff640 1 -- 172.21.15.3:0/3786960486 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbaf4003c20 con 0x7fbafc13c480 2024-03-25T18:26:04.731 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.729+0000 7fbae3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:04.731 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.729+0000 7fbae3fff640 1 -- 172.21.15.3:0/3786960486 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fbaf4003fb0 con 0x7fbafc13c480 2024-03-25T18:26:04.731 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.729+0000 7fbae3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:04.731 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.729+0000 7fbae3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:04.731 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.729+0000 7fbae3fff640 10 monclient: dump: 2024-03-25T18:26:04.731 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:04.731 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:04.731 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:04.731 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:04.731 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:04.731 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:04.731 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:04.731 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:04.731 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:04.732 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:04.732 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.729+0000 7fbaf9ffb640 4 set_mon_vals no callback set 2024-03-25T18:26:04.732 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.730+0000 7fbae3fff640 1 -- 172.21.15.3:0/3786960486 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7fbaf4004190 con 0x7fbafc13c480 2024-03-25T18:26:04.732 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.730+0000 7fbafb7fe640 10 monclient: get_auth_request con 0x7fbae806dd90 auth_method 0 2024-03-25T18:26:04.732 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.731+0000 7fbae3fff640 10 client.4246.objecter ms_handle_connect 0x7fbae806dd90 2024-03-25T18:26:04.736 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.735+0000 7fbae3fff640 1 -- 172.21.15.3:0/3786960486 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7fbaf4054be0 con 0x7fbafc13c480 2024-03-25T18:26:04.736 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.735+0000 7fbae3fff640 10 client.4246.objecter ms_dispatch 0x7fbafc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:04.736 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.735+0000 7fbae3fff640 3 client.4246.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:04.736 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.735+0000 7fbae3fff640 3 client.4246.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:04.736 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.735+0000 7fbae3fff640 20 client.4246.objecter dump_active .. 0 homeless 2024-03-25T18:26:04.737 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.736+0000 7fbae3fff640 1 -- 172.21.15.3:0/3786960486 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbaf40506f0 con 0x7fbafc13c480 2024-03-25T18:26:04.737 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.736+0000 7fbae3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:04.737 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.736+0000 7fbae3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:04.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.993+0000 7fbb02086640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:04.995 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.993+0000 7fbb02086640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:04.995 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.993+0000 7fbb02086640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:26:04.995 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:04.993+0000 7fbb02086640 1 -- 172.21.15.3:0/3786960486 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbafc0aefb0 con 0x7fbafc13c480 2024-03-25T18:26:05.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:05.018+0000 7fbae3fff640 1 -- 172.21.15.3:0/3786960486 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7fbaf4052620 con 0x7fbafc13c480 2024-03-25T18:26:05.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:05.018+0000 7fbae3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:05.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:05.018+0000 7fbae3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:05.019 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:26:05.019 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:05.023 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:05.021+0000 7fbb02086640 10 librados: watch_flush enter 2024-03-25T18:26:05.023 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:05.021+0000 7fbb02086640 10 librados: watch_flush exit 2024-03-25T18:26:05.023 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:05.021+0000 7fbb02086640 20 client.4246.objecter shutdown clearing up homeless session... 2024-03-25T18:26:05.023 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:05.021+0000 7fbb02086640 10 client.4246.objecter successfully canceled tick 2024-03-25T18:26:05.023 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:05.021+0000 7fbb02086640 1 -- 172.21.15.3:0/3786960486 >> v1:172.21.15.3:6800/4195140966 conn(0x7fbae806dd90 legacy=0x7fbae8070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:05.023 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:05.021+0000 7fbb02086640 10 monclient: shutdown 2024-03-25T18:26:05.023 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:05.021+0000 7fbb02086640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:05.023 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:05.021+0000 7fbb02086640 1 -- 172.21.15.3:0/3786960486 >> v1:172.21.15.16:6789/0 conn(0x7fbafc13c480 legacy=0x7fbafc1e2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:05.023 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:05.021+0000 7fbb02086640 1 -- 172.21.15.3:0/3786960486 shutdown_connections 2024-03-25T18:26:05.023 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:05.021+0000 7fbb02086640 1 -- 172.21.15.3:0/3786960486 >> 172.21.15.3:0/3786960486 conn(0x7fbafc0617a0 msgr2=0x7fbafc1405c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:05.023 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:05.021+0000 7fbb02086640 1 -- 172.21.15.3:0/3786960486 shutdown_connections 2024-03-25T18:26:05.023 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:05.021+0000 7fbb02086640 1 -- 172.21.15.3:0/3786960486 wait complete. 2024-03-25T18:26:05.023 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:05.021+0000 7fbb02086640 1 librados: shutdown 2024-03-25T18:26:05.034 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:11.842 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:11.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:11.956+0000 7f0e7134d640 10 monclient: get_monmap_and_config 2024-03-25T18:26:11.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:11.956+0000 7f0e7134d640 10 monclient: build_initial_monmap 2024-03-25T18:26:11.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:11.956+0000 7f0e7134d640 10 monclient: monmap: 2024-03-25T18:26:11.958 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:26:11.958 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:11.958 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:26:11.957901+0000 2024-03-25T18:26:11.958 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:26:11.957901+0000 2024-03-25T18:26:11.958 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:11.958 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:11.958 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:26:11.959 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:26:11.959 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:26:11.959 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:11.960 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:11.957+0000 7f0e7134d640 1 Processor -- start 2024-03-25T18:26:11.960 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:11.957+0000 7f0e7134d640 1 -- start start 2024-03-25T18:26:11.960 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:11.957+0000 7f0e7134d640 10 monclient: init 2024-03-25T18:26:11.960 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:11.958+0000 7f0e7134d640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:11.960 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:11.958+0000 7f0e7134d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:11.960 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:11.958+0000 7f0e7134d640 10 monclient(hunting): picked mon.noname-a con 0x7f0e6c0b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:26:11.960 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:11.958+0000 7f0e7134d640 10 monclient(hunting): picked mon.noname-c con 0x7f0e6c0b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:26:11.960 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:11.958+0000 7f0e7134d640 10 monclient(hunting): picked mon.noname-b con 0x7f0e6c13c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:26:11.960 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:11.958+0000 7f0e7134d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e6c0ab0e0 con 0x7f0e6c0b4ff0 2024-03-25T18:26:11.960 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:11.958+0000 7f0e7134d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e6c0ab3c0 con 0x7f0e6c0b4670 2024-03-25T18:26:11.960 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:11.958+0000 7f0e7134d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e6c0a9f30 con 0x7f0e6c13c480 2024-03-25T18:26:11.960 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:11.958+0000 7f0e7134d640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:11.961 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:11.958+0000 7f0e7134d640 10 monclient(hunting): authenticate will time out at 7426.218838s 2024-03-25T18:26:11.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:11.969+0000 7f0e6a7fc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0e6c0b4670 0x7f0e6c1425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:34316/0 (socket says 172.21.15.3:34316) 2024-03-25T18:26:11.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:11.969+0000 7f0e6a7fc640 1 -- 172.21.15.3:0/3125252728 learned_addr learned my addr 172.21.15.3:0/3125252728 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:26:11.991 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:11.989+0000 7f0e69ffb640 1 -- 172.21.15.3:0/3125252728 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2875547604 0 0) 0x7f0e6c0ab3c0 con 0x7f0e6c0b4670 2024-03-25T18:26:11.991 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:11.989+0000 7f0e69ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:11.991 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:11.989+0000 7f0e69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:11.991 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:11.989+0000 7f0e69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:11.991 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:11.989+0000 7f0e69ffb640 10 monclient(hunting): my global_id is 4249 2024-03-25T18:26:11.991 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:11.989+0000 7f0e69ffb640 1 -- 172.21.15.3:0/3125252728 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e54003520 con 0x7f0e6c0b4670 2024-03-25T18:26:11.998 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:11.997+0000 7f0e69ffb640 1 -- 172.21.15.3:0/3125252728 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3535907560 0 0) 0x7f0e6c0a9f30 con 0x7f0e6c13c480 2024-03-25T18:26:11.998 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:11.997+0000 7f0e69ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:11.999 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:11.997+0000 7f0e69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:11.999 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:11.997+0000 7f0e69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:11.999 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:11.997+0000 7f0e69ffb640 10 monclient(hunting): my global_id is 4256 2024-03-25T18:26:11.999 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:11.997+0000 7f0e69ffb640 1 -- 172.21.15.3:0/3125252728 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e6c0ab3c0 con 0x7f0e6c13c480 2024-03-25T18:26:12.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.001+0000 7f0e69ffb640 1 -- 172.21.15.3:0/3125252728 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1617154249 0 0) 0x7f0e54003520 con 0x7f0e6c0b4670 2024-03-25T18:26:12.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.001+0000 7f0e69ffb640 1 -- 172.21.15.3:0/3125252728 >> v1:172.21.15.3:6790/0 conn(0x7f0e6c13c480 legacy=0x7f0e6c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:12.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.001+0000 7f0e69ffb640 1 -- 172.21.15.3:0/3125252728 >> v1:172.21.15.3:6789/0 conn(0x7f0e6c0b4ff0 legacy=0x7f0e6c13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:12.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.001+0000 7f0e69ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:26:12.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.001+0000 7f0e69ffb640 1 monclient: found mon.noname-c 2024-03-25T18:26:12.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.001+0000 7f0e69ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:12.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.001+0000 7f0e69ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:26:12.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.001+0000 7f0e69ffb640 1 -- 172.21.15.3:0/3125252728 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e6c0a9b70 con 0x7f0e6c0b4670 2024-03-25T18:26:12.004 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.001+0000 7f0e69ffb640 10 monclient: _finish_auth 0 2024-03-25T18:26:12.004 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.001+0000 7f0e69ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:26:12.004 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.001+0000 7f0e69ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:12.004 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.001+0000 7f0e69ffb640 1 -- 172.21.15.3:0/3125252728 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f0e60001690 con 0x7f0e6c0b4670 2024-03-25T18:26:12.004 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.001+0000 7f0e7134d640 5 monclient: authenticate success, global_id 4249 2024-03-25T18:26:12.004 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.001+0000 7f0e69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:12.004 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.001+0000 7f0e69ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:12.004 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.001+0000 7f0e69ffb640 10 monclient: dump: 2024-03-25T18:26:12.004 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:12.004 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:12.004 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:12.004 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:12.004 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:12.004 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:12.005 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:12.005 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:12.005 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:12.005 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:12.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.001+0000 7f0e7134d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:12.015 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.014+0000 7f0e69ffb640 1 -- 172.21.15.3:0/3125252728 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e600017f0 con 0x7f0e6c0b4670 2024-03-25T18:26:12.015 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.014+0000 7f0e69ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:12.015 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.014+0000 7f0e69ffb640 1 -- 172.21.15.3:0/3125252728 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f0e60001c00 con 0x7f0e6c0b4670 2024-03-25T18:26:12.015 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.014+0000 7f0e7134d640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:12.016 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.014+0000 7f0e7134d640 4 set_mon_vals no callback set 2024-03-25T18:26:12.016 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.014+0000 7f0e69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:12.016 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.014+0000 7f0e69ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:12.016 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.014+0000 7f0e69ffb640 10 monclient: dump: 2024-03-25T18:26:12.016 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:12.016 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:12.016 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:12.016 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:12.016 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:12.016 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:12.016 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:12.016 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:12.016 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:12.016 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:12.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.015+0000 7f0e7134d640 10 monclient: shutdown 2024-03-25T18:26:12.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.015+0000 7f0e7134d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:12.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.015+0000 7f0e7134d640 1 -- 172.21.15.3:0/3125252728 >> v1:172.21.15.16:6789/0 conn(0x7f0e6c0b4670 legacy=0x7f0e6c1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:12.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.015+0000 7f0e7134d640 1 -- 172.21.15.3:0/3125252728 shutdown_connections 2024-03-25T18:26:12.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.015+0000 7f0e7134d640 1 -- 172.21.15.3:0/3125252728 >> 172.21.15.3:0/3125252728 conn(0x7f0e6c0617a0 msgr2=0x7f0e6c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:12.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.015+0000 7f0e7134d640 1 -- 172.21.15.3:0/3125252728 shutdown_connections 2024-03-25T18:26:12.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.015+0000 7f0e7134d640 1 -- 172.21.15.3:0/3125252728 wait complete. 2024-03-25T18:26:12.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.016+0000 7f0e7134d640 10 monclient: build_initial_monmap 2024-03-25T18:26:12.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.016+0000 7f0e7134d640 10 monclient: monmap: 2024-03-25T18:26:12.019 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:26:12.019 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:12.019 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:26:12.019 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:26:12.019 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:12.019 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:26:12.019 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:26:12.019 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:26:12.019 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:26:12.019 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:12.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.016+0000 7f0e7134d640 1 librados: starting msgr at 2024-03-25T18:26:12.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.016+0000 7f0e7134d640 1 librados: starting objecter 2024-03-25T18:26:12.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.017+0000 7f0e7134d640 1 Processor -- start 2024-03-25T18:26:12.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.017+0000 7f0e7134d640 1 -- start start 2024-03-25T18:26:12.020 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.017+0000 7f0e7134d640 1 librados: setting wanted keys 2024-03-25T18:26:12.020 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.017+0000 7f0e7134d640 1 librados: calling monclient init 2024-03-25T18:26:12.020 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.017+0000 7f0e7134d640 10 monclient: init 2024-03-25T18:26:12.021 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.018+0000 7f0e7134d640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:12.021 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.018+0000 7f0e7134d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:26:12.021 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.018+0000 7f0e7134d640 10 monclient(hunting): picked mon.noname-b con 0x7f0e6c0b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:26:12.021 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.018+0000 7f0e7134d640 10 monclient(hunting): picked mon.noname-a con 0x7f0e6c0b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:26:12.021 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.018+0000 7f0e7134d640 10 monclient(hunting): picked mon.noname-c con 0x7f0e6c13c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:26:12.021 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.018+0000 7f0e7134d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e6c144a50 con 0x7f0e6c0b4ff0 2024-03-25T18:26:12.021 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.018+0000 7f0e7134d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e6c13ec30 con 0x7f0e6c0b4670 2024-03-25T18:26:12.021 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.018+0000 7f0e7134d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e6c13edc0 con 0x7f0e6c13c480 2024-03-25T18:26:12.021 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.018+0000 7f0e7134d640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:12.021 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.018+0000 7f0e7134d640 10 monclient(hunting): authenticate will time out at 7426.278738s 2024-03-25T18:26:12.039 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.038+0000 7f0e6a7fc640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f0e6c0b4ff0 0x7f0e6c145eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:46714/0 (socket says 172.21.15.3:46714) 2024-03-25T18:26:12.039 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.038+0000 7f0e6a7fc640 1 -- 172.21.15.3:0/4182685274 learned_addr learned my addr 172.21.15.3:0/4182685274 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:26:12.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.039+0000 7f0e537fe640 10 client.?.objecter ms_handle_connect 0x7f0e6c0b4ff0 2024-03-25T18:26:12.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.039+0000 7f0e537fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:12.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.047+0000 7f0e537fe640 10 client.?.objecter ms_handle_connect 0x7f0e6c13c480 2024-03-25T18:26:12.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.047+0000 7f0e537fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:12.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.050+0000 7f0e537fe640 1 -- 172.21.15.3:0/4182685274 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2941307983 0 0) 0x7f0e6c144a50 con 0x7f0e6c0b4ff0 2024-03-25T18:26:12.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.050+0000 7f0e537fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:12.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.050+0000 7f0e537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:12.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.050+0000 7f0e537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:12.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.050+0000 7f0e537fe640 10 monclient(hunting): my global_id is 4287 2024-03-25T18:26:12.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.050+0000 7f0e537fe640 1 -- 172.21.15.3:0/4182685274 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e58004eb0 con 0x7f0e6c0b4ff0 2024-03-25T18:26:12.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.050+0000 7f0e537fe640 10 client.?.objecter ms_handle_connect 0x7f0e6c0b4670 2024-03-25T18:26:12.052 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.050+0000 7f0e537fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:12.066 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.064+0000 7f0e537fe640 1 -- 172.21.15.3:0/4182685274 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 350535310 0 0) 0x7f0e6c13ec30 con 0x7f0e6c0b4670 2024-03-25T18:26:12.066 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.064+0000 7f0e537fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:12.066 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.064+0000 7f0e537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:12.066 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.064+0000 7f0e537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:12.066 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.064+0000 7f0e537fe640 10 monclient(hunting): my global_id is 4252 2024-03-25T18:26:12.066 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.064+0000 7f0e537fe640 1 -- 172.21.15.3:0/4182685274 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e6c144a50 con 0x7f0e6c0b4670 2024-03-25T18:26:12.066 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.064+0000 7f0e537fe640 1 -- 172.21.15.3:0/4182685274 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1621581520 0 0) 0x7f0e6c13edc0 con 0x7f0e6c13c480 2024-03-25T18:26:12.066 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.064+0000 7f0e537fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:12.066 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.064+0000 7f0e537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:12.066 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.064+0000 7f0e537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:12.066 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.064+0000 7f0e537fe640 10 monclient(hunting): my global_id is 4259 2024-03-25T18:26:12.067 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.064+0000 7f0e537fe640 1 -- 172.21.15.3:0/4182685274 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e6c13ec30 con 0x7f0e6c13c480 2024-03-25T18:26:12.087 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.086+0000 7f0e537fe640 1 -- 172.21.15.3:0/4182685274 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2756377658 0 0) 0x7f0e58004eb0 con 0x7f0e6c0b4ff0 2024-03-25T18:26:12.087 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.086+0000 7f0e537fe640 1 -- 172.21.15.3:0/4182685274 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e6c13edc0 con 0x7f0e6c0b4ff0 2024-03-25T18:26:12.087 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.086+0000 7f0e537fe640 1 -- 172.21.15.3:0/4182685274 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f0e60001c00 con 0x7f0e6c0b4ff0 2024-03-25T18:26:12.087 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.086+0000 7f0e537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:12.087 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.086+0000 7f0e537fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:12.087 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.086+0000 7f0e537fe640 10 monclient(hunting): dump: 2024-03-25T18:26:12.087 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:12.087 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:12.088 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:12.088 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:12.088 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:12.088 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:12.088 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:12.088 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:12.088 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:12.088 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:12.088 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.086+0000 7f0e537fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:12.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.090+0000 7f0e537fe640 1 -- 172.21.15.3:0/4182685274 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2024759485 0 0) 0x7f0e6c144a50 con 0x7f0e6c0b4670 2024-03-25T18:26:12.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.090+0000 7f0e537fe640 1 -- 172.21.15.3:0/4182685274 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e58004eb0 con 0x7f0e6c0b4670 2024-03-25T18:26:12.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.090+0000 7f0e537fe640 1 -- 172.21.15.3:0/4182685274 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f0e5c0018a0 con 0x7f0e6c0b4670 2024-03-25T18:26:12.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.090+0000 7f0e537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:12.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.090+0000 7f0e537fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:12.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.090+0000 7f0e537fe640 10 monclient(hunting): dump: 2024-03-25T18:26:12.092 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:12.092 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:12.092 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:12.092 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:12.092 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:12.092 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:12.092 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:12.092 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:12.092 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:12.092 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:12.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.100+0000 7f0e537fe640 1 -- 172.21.15.3:0/4182685274 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2487417679 0 0) 0x7f0e6c13edc0 con 0x7f0e6c0b4ff0 2024-03-25T18:26:12.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.100+0000 7f0e537fe640 1 -- 172.21.15.3:0/4182685274 >> v1:172.21.15.3:6790/0 conn(0x7f0e6c13c480 legacy=0x7f0e6c1e26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:12.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.100+0000 7f0e537fe640 1 -- 172.21.15.3:0/4182685274 >> v1:172.21.15.16:6789/0 conn(0x7f0e6c0b4670 legacy=0x7f0e6c1457a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:12.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.100+0000 7f0e537fe640 10 monclient: _finish_hunting 0 2024-03-25T18:26:12.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.100+0000 7f0e537fe640 1 monclient: found mon.a 2024-03-25T18:26:12.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.100+0000 7f0e537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:12.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.100+0000 7f0e537fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:26:12.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.100+0000 7f0e537fe640 1 -- 172.21.15.3:0/4182685274 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e6c13f6e0 con 0x7f0e6c0b4ff0 2024-03-25T18:26:12.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.100+0000 7f0e537fe640 10 monclient: _finish_auth 0 2024-03-25T18:26:12.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.100+0000 7f0e537fe640 10 monclient: _check_auth_tickets 2024-03-25T18:26:12.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.100+0000 7f0e537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:12.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.100+0000 7f0e7134d640 5 monclient: authenticate success, global_id 4287 2024-03-25T18:26:12.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.100+0000 7f0e7134d640 10 monclient: _renew_subs 2024-03-25T18:26:12.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.100+0000 7f0e7134d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:26:12.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.100+0000 7f0e7134d640 1 -- 172.21.15.3:0/4182685274 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e6c13f930 con 0x7f0e6c0b4ff0 2024-03-25T18:26:12.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.100+0000 7f0e7134d640 10 client.4287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:12.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.100+0000 7f0e7134d640 10 monclient: _renew_subs 2024-03-25T18:26:12.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.100+0000 7f0e7134d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:26:12.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.100+0000 7f0e7134d640 1 -- 172.21.15.3:0/4182685274 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e6c143ab0 con 0x7f0e6c0b4ff0 2024-03-25T18:26:12.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.100+0000 7f0e7134d640 1 librados: init done 2024-03-25T18:26:12.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.101+0000 7f0e7134d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:12.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.101+0000 7f0e7134d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:12.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.101+0000 7f0e7134d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:26:12.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.101+0000 7f0e7134d640 1 -- 172.21.15.3:0/4182685274 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e6c145150 con 0x7f0e6c0b4ff0 2024-03-25T18:26:12.436 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.434+0000 7f0e537fe640 1 -- 172.21.15.3:0/4182685274 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e60004a90 con 0x7f0e6c0b4ff0 2024-03-25T18:26:12.436 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.434+0000 7f0e537fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:12.436 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.434+0000 7f0e537fe640 1 -- 172.21.15.3:0/4182685274 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f0e60004c80 con 0x7f0e6c0b4ff0 2024-03-25T18:26:12.436 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.434+0000 7f0e537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:12.437 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.434+0000 7f0e537fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:12.437 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.434+0000 7f0e537fe640 10 monclient: dump: 2024-03-25T18:26:12.437 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:12.437 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:12.437 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:12.437 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:12.437 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:12.437 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:12.437 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:12.437 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:12.438 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:12.438 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:12.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.434+0000 7f0e697fa640 4 set_mon_vals no callback set 2024-03-25T18:26:12.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.435+0000 7f0e537fe640 1 -- 172.21.15.3:0/4182685274 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f0e60004ea0 con 0x7f0e6c0b4ff0 2024-03-25T18:26:12.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.436+0000 7f0e6affd640 10 monclient: get_auth_request con 0x7f0e5806df70 auth_method 0 2024-03-25T18:26:12.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.436+0000 7f0e537fe640 10 client.4287.objecter ms_handle_connect 0x7f0e5806df70 2024-03-25T18:26:12.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.445+0000 7f0e537fe640 1 -- 172.21.15.3:0/4182685274 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f0e60054820 con 0x7f0e6c0b4ff0 2024-03-25T18:26:12.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.445+0000 7f0e537fe640 10 client.4287.objecter ms_dispatch 0x7f0e6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:12.447 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.445+0000 7f0e537fe640 3 client.4287.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:12.447 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.445+0000 7f0e537fe640 3 client.4287.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:12.447 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.445+0000 7f0e537fe640 20 client.4287.objecter dump_active .. 0 homeless 2024-03-25T18:26:12.447 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.445+0000 7f0e537fe640 1 -- 172.21.15.3:0/4182685274 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0e60050330 con 0x7f0e6c0b4ff0 2024-03-25T18:26:12.447 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.445+0000 7f0e537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:12.447 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.445+0000 7f0e537fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:12.711 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.709+0000 7f0e7134d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:12.711 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.709+0000 7f0e7134d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:12.711 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.709+0000 7f0e7134d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:26:12.711 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.709+0000 7f0e7134d640 1 -- 172.21.15.3:0/4182685274 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e6c06dbd0 con 0x7f0e6c0b4ff0 2024-03-25T18:26:12.732 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.731+0000 7f0e537fe640 1 -- 172.21.15.3:0/4182685274 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f0e60052260 con 0x7f0e6c0b4ff0 2024-03-25T18:26:12.732 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.731+0000 7f0e537fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:12.732 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.731+0000 7f0e537fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:12.732 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:26:12.732 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:12.734 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.732+0000 7f0e7134d640 10 librados: watch_flush enter 2024-03-25T18:26:12.734 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.733+0000 7f0e7134d640 10 librados: watch_flush exit 2024-03-25T18:26:12.734 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.733+0000 7f0e7134d640 20 client.4287.objecter shutdown clearing up homeless session... 2024-03-25T18:26:12.734 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.733+0000 7f0e7134d640 10 client.4287.objecter successfully canceled tick 2024-03-25T18:26:12.735 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.733+0000 7f0e7134d640 1 -- 172.21.15.3:0/4182685274 >> v1:172.21.15.3:6800/4195140966 conn(0x7f0e5806df70 legacy=0x7f0e58070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:12.735 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.733+0000 7f0e7134d640 10 monclient: shutdown 2024-03-25T18:26:12.735 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.733+0000 7f0e7134d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:12.735 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.733+0000 7f0e7134d640 1 -- 172.21.15.3:0/4182685274 >> v1:172.21.15.3:6789/0 conn(0x7f0e6c0b4ff0 legacy=0x7f0e6c145eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:12.735 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.733+0000 7f0e7134d640 1 -- 172.21.15.3:0/4182685274 shutdown_connections 2024-03-25T18:26:12.735 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.733+0000 7f0e7134d640 1 -- 172.21.15.3:0/4182685274 >> 172.21.15.3:0/4182685274 conn(0x7f0e6c0617a0 msgr2=0x7f0e6c1407f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:12.735 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.733+0000 7f0e7134d640 1 -- 172.21.15.3:0/4182685274 shutdown_connections 2024-03-25T18:26:12.735 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.733+0000 7f0e7134d640 1 -- 172.21.15.3:0/4182685274 wait complete. 2024-03-25T18:26:12.736 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:12.733+0000 7f0e7134d640 1 librados: shutdown 2024-03-25T18:26:12.746 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:19.553 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:19.668 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.666+0000 7fd7c3a14640 10 monclient: get_monmap_and_config 2024-03-25T18:26:19.668 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.666+0000 7fd7c3a14640 10 monclient: build_initial_monmap 2024-03-25T18:26:19.668 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.666+0000 7fd7c3a14640 10 monclient: monmap: 2024-03-25T18:26:19.668 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:26:19.668 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:19.668 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:26:19.668012+0000 2024-03-25T18:26:19.668 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:26:19.668012+0000 2024-03-25T18:26:19.668 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:19.668 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:19.668 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:26:19.669 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:26:19.669 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:26:19.669 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:19.670 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.667+0000 7fd7c3a14640 1 Processor -- start 2024-03-25T18:26:19.670 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.667+0000 7fd7c3a14640 1 -- start start 2024-03-25T18:26:19.670 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.667+0000 7fd7c3a14640 10 monclient: init 2024-03-25T18:26:19.670 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.668+0000 7fd7c3a14640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:19.670 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.668+0000 7fd7c3a14640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:26:19.671 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.668+0000 7fd7c3a14640 10 monclient(hunting): picked mon.noname-c con 0x7fd7bc0b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:26:19.671 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.668+0000 7fd7c3a14640 10 monclient(hunting): picked mon.noname-a con 0x7fd7bc0b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:26:19.671 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.668+0000 7fd7c3a14640 10 monclient(hunting): picked mon.noname-b con 0x7fd7bc13c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:26:19.671 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.668+0000 7fd7c3a14640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7bc0ab0e0 con 0x7fd7bc0b4670 2024-03-25T18:26:19.671 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.668+0000 7fd7c3a14640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7bc0ab3c0 con 0x7fd7bc0b4ff0 2024-03-25T18:26:19.671 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.668+0000 7fd7c3a14640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7bc0a9f30 con 0x7fd7bc13c480 2024-03-25T18:26:19.671 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.668+0000 7fd7c3a14640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:19.671 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.668+0000 7fd7c3a14640 10 monclient(hunting): authenticate will time out at 7433.929161s 2024-03-25T18:26:19.675 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.674+0000 7fd7c17b0640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd7bc0b4ff0 0x7fd7bc13bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:50840/0 (socket says 172.21.15.3:50840) 2024-03-25T18:26:19.675 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.674+0000 7fd7c17b0640 1 -- 172.21.15.3:0/1029364290 learned_addr learned my addr 172.21.15.3:0/1029364290 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:26:19.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.757+0000 7fd7b3fff640 1 -- 172.21.15.3:0/1029364290 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3826009411 0 0) 0x7fd7bc0ab0e0 con 0x7fd7bc0b4670 2024-03-25T18:26:19.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.757+0000 7fd7b3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:19.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.757+0000 7fd7b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:19.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.757+0000 7fd7b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:19.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.757+0000 7fd7b3fff640 10 monclient(hunting): my global_id is 4290 2024-03-25T18:26:19.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.757+0000 7fd7b3fff640 1 -- 172.21.15.3:0/1029364290 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7a8003520 con 0x7fd7bc0b4670 2024-03-25T18:26:19.779 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.777+0000 7fd7b3fff640 1 -- 172.21.15.3:0/1029364290 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4136152409 0 0) 0x7fd7a8003520 con 0x7fd7bc0b4670 2024-03-25T18:26:19.779 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.777+0000 7fd7b3fff640 1 -- 172.21.15.3:0/1029364290 >> v1:172.21.15.3:6790/0 conn(0x7fd7bc13c480 legacy=0x7fd7bc14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:19.779 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.777+0000 7fd7b3fff640 1 -- 172.21.15.3:0/1029364290 >> v1:172.21.15.16:6789/0 conn(0x7fd7bc0b4ff0 legacy=0x7fd7bc13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:19.779 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.777+0000 7fd7b3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:26:19.779 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.777+0000 7fd7b3fff640 1 monclient: found mon.noname-a 2024-03-25T18:26:19.779 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.777+0000 7fd7b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:19.779 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.777+0000 7fd7b3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:26:19.779 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.777+0000 7fd7b3fff640 1 -- 172.21.15.3:0/1029364290 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7bc0a9b70 con 0x7fd7bc0b4670 2024-03-25T18:26:19.780 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.777+0000 7fd7b3fff640 10 monclient: _finish_auth 0 2024-03-25T18:26:19.780 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.777+0000 7fd7b3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:26:19.780 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.777+0000 7fd7b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:19.780 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.777+0000 7fd7b3fff640 1 -- 172.21.15.3:0/1029364290 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd7ac001690 con 0x7fd7bc0b4670 2024-03-25T18:26:19.780 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.777+0000 7fd7c3a14640 5 monclient: authenticate success, global_id 4290 2024-03-25T18:26:19.780 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.777+0000 7fd7c3a14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:19.780 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.777+0000 7fd7b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:19.780 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.777+0000 7fd7b3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:19.780 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.777+0000 7fd7b3fff640 10 monclient: dump: 2024-03-25T18:26:19.780 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:19.780 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:19.780 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:19.780 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:19.780 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:19.781 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:19.781 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:19.781 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:19.781 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:19.781 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:19.781 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.777+0000 7fd7c3a14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:19.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.785+0000 7fd7b3fff640 1 -- 172.21.15.3:0/1029364290 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7ac0017f0 con 0x7fd7bc0b4670 2024-03-25T18:26:19.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.785+0000 7fd7b3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:19.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.785+0000 7fd7b3fff640 1 -- 172.21.15.3:0/1029364290 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd7ac001c00 con 0x7fd7bc0b4670 2024-03-25T18:26:19.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.785+0000 7fd7b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:19.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.785+0000 7fd7b3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:19.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.785+0000 7fd7b3fff640 10 monclient: dump: 2024-03-25T18:26:19.786 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:19.786 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:19.786 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:19.786 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:19.786 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:19.786 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:19.787 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:19.787 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:19.787 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:19.787 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:19.787 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.785+0000 7fd7c3a14640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:19.787 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.785+0000 7fd7c3a14640 4 set_mon_vals no callback set 2024-03-25T18:26:19.789 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.785+0000 7fd7c3a14640 10 monclient: shutdown 2024-03-25T18:26:19.789 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.785+0000 7fd7c3a14640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:19.789 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.785+0000 7fd7c3a14640 1 -- 172.21.15.3:0/1029364290 >> v1:172.21.15.3:6789/0 conn(0x7fd7bc0b4670 legacy=0x7fd7bc1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:19.789 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.785+0000 7fd7c3a14640 1 -- 172.21.15.3:0/1029364290 shutdown_connections 2024-03-25T18:26:19.789 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.785+0000 7fd7c3a14640 1 -- 172.21.15.3:0/1029364290 >> 172.21.15.3:0/1029364290 conn(0x7fd7bc0617a0 msgr2=0x7fd7bc061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:19.789 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.786+0000 7fd7c3a14640 1 -- 172.21.15.3:0/1029364290 shutdown_connections 2024-03-25T18:26:19.789 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.786+0000 7fd7c3a14640 1 -- 172.21.15.3:0/1029364290 wait complete. 2024-03-25T18:26:19.790 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.786+0000 7fd7c3a14640 10 monclient: build_initial_monmap 2024-03-25T18:26:19.790 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.786+0000 7fd7c3a14640 10 monclient: monmap: 2024-03-25T18:26:19.790 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:26:19.790 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:19.790 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:26:19.790 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:26:19.790 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:19.790 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:26:19.790 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:26:19.790 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:26:19.790 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:26:19.790 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:19.790 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.787+0000 7fd7c3a14640 1 librados: starting msgr at 2024-03-25T18:26:19.790 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.787+0000 7fd7c3a14640 1 librados: starting objecter 2024-03-25T18:26:19.791 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.787+0000 7fd7c3a14640 1 Processor -- start 2024-03-25T18:26:19.791 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.787+0000 7fd7c3a14640 1 -- start start 2024-03-25T18:26:19.791 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.787+0000 7fd7c3a14640 1 librados: setting wanted keys 2024-03-25T18:26:19.791 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.787+0000 7fd7c3a14640 1 librados: calling monclient init 2024-03-25T18:26:19.791 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.787+0000 7fd7c3a14640 10 monclient: init 2024-03-25T18:26:19.791 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.787+0000 7fd7c3a14640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:19.791 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.787+0000 7fd7c3a14640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:26:19.791 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.787+0000 7fd7c3a14640 10 monclient(hunting): picked mon.noname-c con 0x7fd7bc0b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:26:19.791 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.787+0000 7fd7c3a14640 10 monclient(hunting): picked mon.noname-b con 0x7fd7bc0b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:26:19.791 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.787+0000 7fd7c3a14640 10 monclient(hunting): picked mon.noname-a con 0x7fd7bc13c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:26:19.791 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.787+0000 7fd7c3a14640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7bc1dbf50 con 0x7fd7bc13c480 2024-03-25T18:26:19.791 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.787+0000 7fd7c3a14640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7bc144650 con 0x7fd7bc0b4ff0 2024-03-25T18:26:19.791 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.787+0000 7fd7c3a14640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7bc144930 con 0x7fd7bc0b4670 2024-03-25T18:26:19.792 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.787+0000 7fd7c3a14640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:19.792 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.787+0000 7fd7c3a14640 10 monclient(hunting): authenticate will time out at 7434.048316s 2024-03-25T18:26:19.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.804+0000 7fd7c1fb1640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd7bc13c480 0x7fd7bc1db840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:33938/0 (socket says 172.21.15.3:33938) 2024-03-25T18:26:19.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.804+0000 7fd7c1fb1640 1 -- 172.21.15.3:0/908465275 learned_addr learned my addr 172.21.15.3:0/908465275 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:26:19.806 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.805+0000 7fd7b1ffb640 10 client.?.objecter ms_handle_connect 0x7fd7bc13c480 2024-03-25T18:26:19.806 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.805+0000 7fd7b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:19.818 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.816+0000 7fd7b1ffb640 10 client.?.objecter ms_handle_connect 0x7fd7bc0b4ff0 2024-03-25T18:26:19.818 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.816+0000 7fd7b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:19.818 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.817+0000 7fd7b1ffb640 10 client.?.objecter ms_handle_connect 0x7fd7bc0b4670 2024-03-25T18:26:19.818 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.817+0000 7fd7b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:19.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.826+0000 7fd7b1ffb640 1 -- 172.21.15.3:0/908465275 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153870406 0 0) 0x7fd7bc144650 con 0x7fd7bc0b4ff0 2024-03-25T18:26:19.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.826+0000 7fd7b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:19.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.826+0000 7fd7b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:19.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.826+0000 7fd7b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:19.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.826+0000 7fd7b1ffb640 10 monclient(hunting): my global_id is 4258 2024-03-25T18:26:19.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.826+0000 7fd7b1ffb640 1 -- 172.21.15.3:0/908465275 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7a0004eb0 con 0x7fd7bc0b4ff0 2024-03-25T18:26:19.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.829+0000 7fd7b1ffb640 1 -- 172.21.15.3:0/908465275 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3783428596 0 0) 0x7fd7bc144930 con 0x7fd7bc0b4670 2024-03-25T18:26:19.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.829+0000 7fd7b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:19.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.829+0000 7fd7b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:19.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.829+0000 7fd7b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:19.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.829+0000 7fd7b1ffb640 10 monclient(hunting): my global_id is 4265 2024-03-25T18:26:19.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.829+0000 7fd7b1ffb640 1 -- 172.21.15.3:0/908465275 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7bc144650 con 0x7fd7bc0b4670 2024-03-25T18:26:19.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.851+0000 7fd7b1ffb640 1 -- 172.21.15.3:0/908465275 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765116747 0 0) 0x7fd7bc1dbf50 con 0x7fd7bc13c480 2024-03-25T18:26:19.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.851+0000 7fd7b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:19.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.851+0000 7fd7b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:19.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.851+0000 7fd7b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:19.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.851+0000 7fd7b1ffb640 10 monclient(hunting): my global_id is 4293 2024-03-25T18:26:19.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.851+0000 7fd7b1ffb640 1 -- 172.21.15.3:0/908465275 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7bc144930 con 0x7fd7bc13c480 2024-03-25T18:26:19.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.851+0000 7fd7b1ffb640 1 -- 172.21.15.3:0/908465275 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2243531464 0 0) 0x7fd7bc144650 con 0x7fd7bc0b4670 2024-03-25T18:26:19.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.851+0000 7fd7b1ffb640 1 -- 172.21.15.3:0/908465275 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7bc1dbf50 con 0x7fd7bc0b4670 2024-03-25T18:26:19.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.851+0000 7fd7b1ffb640 1 -- 172.21.15.3:0/908465275 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd7b80018a0 con 0x7fd7bc0b4670 2024-03-25T18:26:19.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.851+0000 7fd7b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:19.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.851+0000 7fd7b1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:19.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.851+0000 7fd7b1ffb640 10 monclient(hunting): dump: 2024-03-25T18:26:19.854 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:19.854 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:19.854 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:19.854 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:19.854 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:19.854 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:19.854 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:19.854 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:19.854 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:19.854 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:19.854 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.851+0000 7fd7b1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:19.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.856+0000 7fd7b1ffb640 1 -- 172.21.15.3:0/908465275 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3715830196 0 0) 0x7fd7a0004eb0 con 0x7fd7bc0b4ff0 2024-03-25T18:26:19.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.856+0000 7fd7b1ffb640 1 -- 172.21.15.3:0/908465275 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7bc144650 con 0x7fd7bc0b4ff0 2024-03-25T18:26:19.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.856+0000 7fd7b1ffb640 1 -- 172.21.15.3:0/908465275 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd7ac001c00 con 0x7fd7bc0b4ff0 2024-03-25T18:26:19.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.856+0000 7fd7b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:19.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.856+0000 7fd7b1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:19.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.856+0000 7fd7b1ffb640 10 monclient(hunting): dump: 2024-03-25T18:26:19.857 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:19.858 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:19.858 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:19.858 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:19.858 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:19.858 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:19.858 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:19.858 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:19.858 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:19.858 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:19.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.866+0000 7fd7b1ffb640 1 -- 172.21.15.3:0/908465275 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 566535635 0 0) 0x7fd7bc1dbf50 con 0x7fd7bc0b4670 2024-03-25T18:26:19.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.866+0000 7fd7b1ffb640 1 -- 172.21.15.3:0/908465275 >> v1:172.21.15.16:6789/0 conn(0x7fd7bc0b4ff0 legacy=0x7fd7bc1db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:19.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.866+0000 7fd7b1ffb640 1 -- 172.21.15.3:0/908465275 >> v1:172.21.15.3:6789/0 conn(0x7fd7bc13c480 legacy=0x7fd7bc1db840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:19.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.866+0000 7fd7b1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:26:19.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.866+0000 7fd7b1ffb640 1 monclient: found mon.c 2024-03-25T18:26:19.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.866+0000 7fd7b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:19.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.866+0000 7fd7b1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:26:19.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.866+0000 7fd7b1ffb640 1 -- 172.21.15.3:0/908465275 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7bc13ed60 con 0x7fd7bc0b4670 2024-03-25T18:26:19.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.866+0000 7fd7b1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:26:19.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.866+0000 7fd7b1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:26:19.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.866+0000 7fd7b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:19.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.866+0000 7fd7c3a14640 5 monclient: authenticate success, global_id 4265 2024-03-25T18:26:19.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.866+0000 7fd7c3a14640 10 monclient: _renew_subs 2024-03-25T18:26:19.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.866+0000 7fd7c3a14640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:26:19.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.866+0000 7fd7c3a14640 1 -- 172.21.15.3:0/908465275 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7bc145460 con 0x7fd7bc0b4670 2024-03-25T18:26:19.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.866+0000 7fd7c3a14640 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:19.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.866+0000 7fd7c3a14640 10 monclient: _renew_subs 2024-03-25T18:26:19.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.866+0000 7fd7c3a14640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:26:19.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.866+0000 7fd7c3a14640 1 -- 172.21.15.3:0/908465275 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7bc143840 con 0x7fd7bc0b4670 2024-03-25T18:26:19.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.866+0000 7fd7c3a14640 1 librados: init done 2024-03-25T18:26:19.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.867+0000 7fd7c3a14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:19.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.867+0000 7fd7c3a14640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:19.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.867+0000 7fd7c3a14640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:26:19.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:19.867+0000 7fd7c3a14640 1 -- 172.21.15.3:0/908465275 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7bc06dbd0 con 0x7fd7bc0b4670 2024-03-25T18:26:20.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.266+0000 7fd7b1ffb640 1 -- 172.21.15.3:0/908465275 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7b8001cb0 con 0x7fd7bc0b4670 2024-03-25T18:26:20.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.266+0000 7fd7b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:20.268 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.266+0000 7fd7b1ffb640 1 -- 172.21.15.3:0/908465275 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd7b8004980 con 0x7fd7bc0b4670 2024-03-25T18:26:20.268 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.266+0000 7fd7b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:20.268 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.266+0000 7fd7b1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:20.268 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.266+0000 7fd7b1ffb640 10 monclient: dump: 2024-03-25T18:26:20.268 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:20.268 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:20.268 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:20.268 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:20.268 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:20.268 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:20.268 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:20.268 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:20.268 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:20.268 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:20.268 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.266+0000 7fd7b37fe640 4 set_mon_vals no callback set 2024-03-25T18:26:20.269 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.267+0000 7fd7b1ffb640 1 -- 172.21.15.3:0/908465275 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7fd7b8004b60 con 0x7fd7bc0b4670 2024-03-25T18:26:20.269 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.267+0000 7fd7c0faf640 10 monclient: get_auth_request con 0x7fd7a006df70 auth_method 0 2024-03-25T18:26:20.269 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.267+0000 7fd7b1ffb640 10 client.4265.objecter ms_handle_connect 0x7fd7a006df70 2024-03-25T18:26:20.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.275+0000 7fd7b1ffb640 1 -- 172.21.15.3:0/908465275 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7fd7b80546c0 con 0x7fd7bc0b4670 2024-03-25T18:26:20.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.275+0000 7fd7b1ffb640 10 client.4265.objecter ms_dispatch 0x7fd7bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:20.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.275+0000 7fd7b1ffb640 3 client.4265.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:20.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.275+0000 7fd7b1ffb640 3 client.4265.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:20.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.275+0000 7fd7b1ffb640 20 client.4265.objecter dump_active .. 0 homeless 2024-03-25T18:26:20.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.275+0000 7fd7b1ffb640 1 -- 172.21.15.3:0/908465275 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd7b80501d0 con 0x7fd7bc0b4670 2024-03-25T18:26:20.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.275+0000 7fd7b1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:20.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.275+0000 7fd7b1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:20.530 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.529+0000 7fd7c3a14640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:20.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.529+0000 7fd7c3a14640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:20.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.529+0000 7fd7c3a14640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:26:20.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.529+0000 7fd7c3a14640 1 -- 172.21.15.3:0/908465275 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7bc1325a0 con 0x7fd7bc0b4670 2024-03-25T18:26:20.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.624+0000 7fd7b1ffb640 1 -- 172.21.15.3:0/908465275 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7fd7b8052100 con 0x7fd7bc0b4670 2024-03-25T18:26:20.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.625+0000 7fd7b1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:20.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.625+0000 7fd7b1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:20.626 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:26:20.626 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:20.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.627+0000 7fd7c3a14640 10 librados: watch_flush enter 2024-03-25T18:26:20.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.627+0000 7fd7c3a14640 10 librados: watch_flush exit 2024-03-25T18:26:20.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.627+0000 7fd7c3a14640 20 client.4265.objecter shutdown clearing up homeless session... 2024-03-25T18:26:20.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.627+0000 7fd7c3a14640 10 client.4265.objecter successfully canceled tick 2024-03-25T18:26:20.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.627+0000 7fd7c3a14640 1 -- 172.21.15.3:0/908465275 >> v1:172.21.15.3:6800/4195140966 conn(0x7fd7a006df70 legacy=0x7fd7a0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:20.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.627+0000 7fd7c3a14640 10 monclient: shutdown 2024-03-25T18:26:20.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.627+0000 7fd7c3a14640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:20.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.627+0000 7fd7c3a14640 1 -- 172.21.15.3:0/908465275 >> v1:172.21.15.3:6790/0 conn(0x7fd7bc0b4670 legacy=0x7fd7bc144d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:20.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.627+0000 7fd7c3a14640 1 -- 172.21.15.3:0/908465275 shutdown_connections 2024-03-25T18:26:20.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.627+0000 7fd7c3a14640 1 -- 172.21.15.3:0/908465275 >> 172.21.15.3:0/908465275 conn(0x7fd7bc0617a0 msgr2=0x7fd7bc140860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:20.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.627+0000 7fd7c3a14640 1 -- 172.21.15.3:0/908465275 shutdown_connections 2024-03-25T18:26:20.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.627+0000 7fd7c3a14640 1 -- 172.21.15.3:0/908465275 wait complete. 2024-03-25T18:26:20.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:20.628+0000 7fd7c3a14640 1 librados: shutdown 2024-03-25T18:26:20.641 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:20.690 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:01:34.419 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:20.741 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:01:34.471 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:20.792 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:01:34.484 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:20.799 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:01:34.528 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:20.800 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:01:34.529 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:20.800 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:01:34.492 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:20.820 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:01:34.549 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:20.850 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:01:34.542 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:20.864 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:01:34.593 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:27.447 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:27.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.563+0000 7f48b00ed640 10 monclient: get_monmap_and_config 2024-03-25T18:26:27.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.563+0000 7f48b00ed640 10 monclient: build_initial_monmap 2024-03-25T18:26:27.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.563+0000 7f48b00ed640 10 monclient: monmap: 2024-03-25T18:26:27.565 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:26:27.565 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:27.565 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:26:27.564402+0000 2024-03-25T18:26:27.565 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:26:27.564402+0000 2024-03-25T18:26:27.565 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:27.565 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:27.565 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:26:27.565 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:26:27.565 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:26:27.565 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:27.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.564+0000 7f48b00ed640 1 Processor -- start 2024-03-25T18:26:27.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.564+0000 7f48b00ed640 1 -- start start 2024-03-25T18:26:27.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.564+0000 7f48b00ed640 10 monclient: init 2024-03-25T18:26:27.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.565+0000 7f48b00ed640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:27.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.565+0000 7f48b00ed640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:27.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.565+0000 7f48b00ed640 10 monclient(hunting): picked mon.noname-a con 0x7f48a80b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:26:27.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.565+0000 7f48b00ed640 10 monclient(hunting): picked mon.noname-b con 0x7f48a80b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:26:27.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.565+0000 7f48b00ed640 10 monclient(hunting): picked mon.noname-c con 0x7f48a813c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:26:27.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.565+0000 7f48b00ed640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48a80ab0e0 con 0x7f48a80b4ff0 2024-03-25T18:26:27.568 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.565+0000 7f48b00ed640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48a80ab3c0 con 0x7f48a813c480 2024-03-25T18:26:27.568 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.565+0000 7f48b00ed640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48a80a9f30 con 0x7f48a80b4670 2024-03-25T18:26:27.568 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.565+0000 7f48b00ed640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:27.568 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.565+0000 7f48b00ed640 10 monclient(hunting): authenticate will time out at 7441.826010s 2024-03-25T18:26:27.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.570+0000 7f48ad688640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f48a80b4670 0x7f48a81425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:56862/0 (socket says 172.21.15.3:56862) 2024-03-25T18:26:27.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.570+0000 7f48ad688640 1 -- 172.21.15.3:0/161093650 learned_addr learned my addr 172.21.15.3:0/161093650 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:26:27.583 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.582+0000 7f48ace87640 1 -- 172.21.15.3:0/161093650 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721864541 0 0) 0x7f48a80a9f30 con 0x7f48a80b4670 2024-03-25T18:26:27.583 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.582+0000 7f48ace87640 10 monclient(hunting): _negotiate 2024-03-25T18:26:27.583 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.582+0000 7f48ace87640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:27.583 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.582+0000 7f48ace87640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:27.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.582+0000 7f48ace87640 10 monclient(hunting): my global_id is 4268 2024-03-25T18:26:27.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.582+0000 7f48ace87640 1 -- 172.21.15.3:0/161093650 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4894003520 con 0x7f48a80b4670 2024-03-25T18:26:27.603 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.602+0000 7f48ace87640 1 -- 172.21.15.3:0/161093650 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3758777642 0 0) 0x7f48a80ab3c0 con 0x7f48a813c480 2024-03-25T18:26:27.603 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.602+0000 7f48ace87640 10 monclient(hunting): _negotiate 2024-03-25T18:26:27.603 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.602+0000 7f48ace87640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:27.603 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.602+0000 7f48ace87640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:27.603 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.602+0000 7f48ace87640 10 monclient(hunting): my global_id is 4261 2024-03-25T18:26:27.604 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.602+0000 7f48ace87640 1 -- 172.21.15.3:0/161093650 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48a80a9f30 con 0x7f48a813c480 2024-03-25T18:26:27.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.605+0000 7f48ace87640 1 -- 172.21.15.3:0/161093650 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4034206788 0 0) 0x7f4894003520 con 0x7f48a80b4670 2024-03-25T18:26:27.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.605+0000 7f48ace87640 1 -- 172.21.15.3:0/161093650 >> v1:172.21.15.16:6789/0 conn(0x7f48a813c480 legacy=0x7f48a814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:27.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.605+0000 7f48ace87640 1 -- 172.21.15.3:0/161093650 >> v1:172.21.15.3:6789/0 conn(0x7f48a80b4ff0 legacy=0x7f48a813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:27.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.605+0000 7f48ace87640 10 monclient: _finish_hunting 0 2024-03-25T18:26:27.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.605+0000 7f48ace87640 1 monclient: found mon.noname-b 2024-03-25T18:26:27.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.605+0000 7f48ace87640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:27.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.605+0000 7f48ace87640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-25T18:26:27.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.605+0000 7f48ace87640 1 -- 172.21.15.3:0/161093650 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a80a9b70 con 0x7f48a80b4670 2024-03-25T18:26:27.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.605+0000 7f48ace87640 10 monclient: _finish_auth 0 2024-03-25T18:26:27.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.605+0000 7f48ace87640 10 monclient: _check_auth_tickets 2024-03-25T18:26:27.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.605+0000 7f48ace87640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:27.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.605+0000 7f48ace87640 1 -- 172.21.15.3:0/161093650 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4898001690 con 0x7f48a80b4670 2024-03-25T18:26:27.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.605+0000 7f48b00ed640 5 monclient: authenticate success, global_id 4268 2024-03-25T18:26:27.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.605+0000 7f48b00ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:27.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.605+0000 7f48ace87640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:27.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.605+0000 7f48ace87640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:27.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.605+0000 7f48ace87640 10 monclient: dump: 2024-03-25T18:26:27.611 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:27.611 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:27.611 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:27.611 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:27.611 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:27.612 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:27.612 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:27.612 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:27.612 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:27.612 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:27.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.605+0000 7f48b00ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:27.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.613+0000 7f48ace87640 1 -- 172.21.15.3:0/161093650 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48980017f0 con 0x7f48a80b4670 2024-03-25T18:26:27.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.613+0000 7f48ace87640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:27.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.613+0000 7f48ace87640 1 -- 172.21.15.3:0/161093650 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4898001c00 con 0x7f48a80b4670 2024-03-25T18:26:27.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.613+0000 7f48b00ed640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:27.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.613+0000 7f48b00ed640 4 set_mon_vals no callback set 2024-03-25T18:26:27.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.613+0000 7f48ace87640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:27.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.613+0000 7f48ace87640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:27.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.613+0000 7f48ace87640 10 monclient: dump: 2024-03-25T18:26:27.615 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:27.615 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:27.615 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:27.616 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:27.616 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:27.616 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:27.616 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:27.616 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:27.616 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:27.616 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:27.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.614+0000 7f48b00ed640 10 monclient: shutdown 2024-03-25T18:26:27.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.614+0000 7f48b00ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:27.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.614+0000 7f48b00ed640 1 -- 172.21.15.3:0/161093650 >> v1:172.21.15.3:6790/0 conn(0x7f48a80b4670 legacy=0x7f48a81425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:27.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.614+0000 7f48b00ed640 1 -- 172.21.15.3:0/161093650 shutdown_connections 2024-03-25T18:26:27.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.614+0000 7f48b00ed640 1 -- 172.21.15.3:0/161093650 >> 172.21.15.3:0/161093650 conn(0x7f48a80617a0 msgr2=0x7f48a8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:27.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.614+0000 7f48b00ed640 1 -- 172.21.15.3:0/161093650 shutdown_connections 2024-03-25T18:26:27.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.614+0000 7f48b00ed640 1 -- 172.21.15.3:0/161093650 wait complete. 2024-03-25T18:26:27.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.615+0000 7f48b00ed640 10 monclient: build_initial_monmap 2024-03-25T18:26:27.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.615+0000 7f48b00ed640 10 monclient: monmap: 2024-03-25T18:26:27.619 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:26:27.619 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:27.619 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:26:27.619 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:26:27.619 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:27.619 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:26:27.619 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:26:27.620 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:26:27.620 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:26:27.620 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:27.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.615+0000 7f48b00ed640 1 librados: starting msgr at 2024-03-25T18:26:27.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.615+0000 7f48b00ed640 1 librados: starting objecter 2024-03-25T18:26:27.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.615+0000 7f48b00ed640 1 Processor -- start 2024-03-25T18:26:27.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.616+0000 7f48b00ed640 1 -- start start 2024-03-25T18:26:27.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.616+0000 7f48b00ed640 1 librados: setting wanted keys 2024-03-25T18:26:27.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.616+0000 7f48b00ed640 1 librados: calling monclient init 2024-03-25T18:26:27.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.616+0000 7f48b00ed640 10 monclient: init 2024-03-25T18:26:27.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.616+0000 7f48b00ed640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:27.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.616+0000 7f48b00ed640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:26:27.621 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.616+0000 7f48b00ed640 10 monclient(hunting): picked mon.noname-b con 0x7f48a80b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:26:27.621 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.616+0000 7f48b00ed640 10 monclient(hunting): picked mon.noname-a con 0x7f48a80b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:26:27.621 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.617+0000 7f48b00ed640 10 monclient(hunting): picked mon.noname-c con 0x7f48a813c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:26:27.621 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.617+0000 7f48b00ed640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48a80b1980 con 0x7f48a80b4ff0 2024-03-25T18:26:27.621 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.617+0000 7f48b00ed640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48a80b07d0 con 0x7f48a80b4670 2024-03-25T18:26:27.621 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.617+0000 7f48b00ed640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48a80b0ab0 con 0x7f48a813c480 2024-03-25T18:26:27.621 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.617+0000 7f48b00ed640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:27.621 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.617+0000 7f48b00ed640 10 monclient(hunting): authenticate will time out at 7441.877446s 2024-03-25T18:26:27.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.623+0000 7f48ade89640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f48a80b4670 0x7f48a80ae6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:55098/0 (socket says 172.21.15.3:55098) 2024-03-25T18:26:27.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.623+0000 7f48ade89640 1 -- 172.21.15.3:0/3293804631 learned_addr learned my addr 172.21.15.3:0/3293804631 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:26:27.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.625+0000 7f489e7fc640 10 client.?.objecter ms_handle_connect 0x7f48a80b4670 2024-03-25T18:26:27.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.625+0000 7f489e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:27.632 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.630+0000 7f489e7fc640 1 -- 172.21.15.3:0/3293804631 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761111123 0 0) 0x7f48a80b07d0 con 0x7f48a80b4670 2024-03-25T18:26:27.632 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.630+0000 7f489e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:27.632 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.630+0000 7f489e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:27.632 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.630+0000 7f489e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:27.632 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.630+0000 7f489e7fc640 10 monclient(hunting): my global_id is 4264 2024-03-25T18:26:27.632 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.630+0000 7f489e7fc640 1 -- 172.21.15.3:0/3293804631 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f488c004eb0 con 0x7f48a80b4670 2024-03-25T18:26:27.637 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.636+0000 7f489e7fc640 10 client.?.objecter ms_handle_connect 0x7f48a80b4ff0 2024-03-25T18:26:27.637 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.636+0000 7f489e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:27.639 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.638+0000 7f489e7fc640 10 client.?.objecter ms_handle_connect 0x7f48a813c480 2024-03-25T18:26:27.639 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.638+0000 7f489e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:27.644 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.643+0000 7f489e7fc640 1 -- 172.21.15.3:0/3293804631 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2637731101 0 0) 0x7f48a80b1980 con 0x7f48a80b4ff0 2024-03-25T18:26:27.644 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.643+0000 7f489e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:27.644 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.643+0000 7f489e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:27.645 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.643+0000 7f489e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:27.645 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.643+0000 7f489e7fc640 10 monclient(hunting): my global_id is 4299 2024-03-25T18:26:27.645 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.643+0000 7f489e7fc640 1 -- 172.21.15.3:0/3293804631 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48a80b07d0 con 0x7f48a80b4ff0 2024-03-25T18:26:27.648 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.647+0000 7f489e7fc640 1 -- 172.21.15.3:0/3293804631 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3463999390 0 0) 0x7f488c004eb0 con 0x7f48a80b4670 2024-03-25T18:26:27.648 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.647+0000 7f489e7fc640 1 -- 172.21.15.3:0/3293804631 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48a80b1980 con 0x7f48a80b4670 2024-03-25T18:26:27.648 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.647+0000 7f489e7fc640 1 -- 172.21.15.3:0/3293804631 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f48a4001a30 con 0x7f48a80b4670 2024-03-25T18:26:27.648 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.647+0000 7f489e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:27.648 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.647+0000 7f489e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:27.648 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.647+0000 7f489e7fc640 10 monclient(hunting): dump: 2024-03-25T18:26:27.649 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:27.649 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:27.649 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:27.649 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:27.649 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:27.649 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:27.649 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:27.649 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:27.649 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:27.649 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:27.649 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.647+0000 7f489e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:27.657 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.655+0000 7f489e7fc640 1 -- 172.21.15.3:0/3293804631 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2374916077 0 0) 0x7f48a80b07d0 con 0x7f48a80b4ff0 2024-03-25T18:26:27.657 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.655+0000 7f489e7fc640 1 -- 172.21.15.3:0/3293804631 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f488c004eb0 con 0x7f48a80b4ff0 2024-03-25T18:26:27.657 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.655+0000 7f489e7fc640 1 -- 172.21.15.3:0/3293804631 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4898001b00 con 0x7f48a80b4ff0 2024-03-25T18:26:27.657 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.655+0000 7f489e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:27.657 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.655+0000 7f489e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:27.657 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.655+0000 7f489e7fc640 10 monclient(hunting): dump: 2024-03-25T18:26:27.657 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:27.657 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:27.657 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:27.657 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:27.658 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:27.658 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:27.658 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:27.658 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:27.658 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:27.658 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:27.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.655+0000 7f489e7fc640 1 -- 172.21.15.3:0/3293804631 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2486513033 0 0) 0x7f48a80b0ab0 con 0x7f48a813c480 2024-03-25T18:26:27.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.656+0000 7f489e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:27.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.656+0000 7f489e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:27.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.656+0000 7f489e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:27.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.656+0000 7f489e7fc640 10 monclient(hunting): my global_id is 4271 2024-03-25T18:26:27.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.656+0000 7f489e7fc640 1 -- 172.21.15.3:0/3293804631 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48a80b07d0 con 0x7f48a813c480 2024-03-25T18:26:27.665 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.664+0000 7f489e7fc640 1 -- 172.21.15.3:0/3293804631 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 425166903 0 0) 0x7f488c004eb0 con 0x7f48a80b4ff0 2024-03-25T18:26:27.665 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.664+0000 7f489e7fc640 1 -- 172.21.15.3:0/3293804631 >> v1:172.21.15.3:6790/0 conn(0x7f48a813c480 legacy=0x7f48a81e2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:27.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.664+0000 7f489e7fc640 1 -- 172.21.15.3:0/3293804631 >> v1:172.21.15.16:6789/0 conn(0x7f48a80b4670 legacy=0x7f48a80ae6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:27.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.664+0000 7f489e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:26:27.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.664+0000 7f489e7fc640 1 monclient: found mon.a 2024-03-25T18:26:27.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.664+0000 7f489e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:27.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.664+0000 7f489e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:26:27.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.664+0000 7f489e7fc640 1 -- 172.21.15.3:0/3293804631 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a80b1460 con 0x7f48a80b4ff0 2024-03-25T18:26:27.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.664+0000 7f489e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:26:27.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.664+0000 7f489e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:26:27.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.664+0000 7f489e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:27.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.664+0000 7f48b00ed640 5 monclient: authenticate success, global_id 4299 2024-03-25T18:26:27.667 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.664+0000 7f48b00ed640 10 monclient: _renew_subs 2024-03-25T18:26:27.667 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.664+0000 7f48b00ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:26:27.667 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.664+0000 7f48b00ed640 1 -- 172.21.15.3:0/3293804631 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48a80b0ed0 con 0x7f48a80b4ff0 2024-03-25T18:26:27.667 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.664+0000 7f48b00ed640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:27.667 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.664+0000 7f48b00ed640 10 monclient: _renew_subs 2024-03-25T18:26:27.667 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.664+0000 7f48b00ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:26:27.667 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.664+0000 7f48b00ed640 1 -- 172.21.15.3:0/3293804631 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f48a80af590 con 0x7f48a80b4ff0 2024-03-25T18:26:27.667 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.664+0000 7f48b00ed640 1 librados: init done 2024-03-25T18:26:27.668 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.665+0000 7f48b00ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:27.668 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.665+0000 7f48b00ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:27.668 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.665+0000 7f48b00ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:26:27.668 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.665+0000 7f48b00ed640 1 -- 172.21.15.3:0/3293804631 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48a806dbd0 con 0x7f48a80b4ff0 2024-03-25T18:26:27.986 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.985+0000 7f489e7fc640 1 -- 172.21.15.3:0/3293804631 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4898001ca0 con 0x7f48a80b4ff0 2024-03-25T18:26:27.987 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.985+0000 7f489e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:27.987 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.985+0000 7f489e7fc640 1 -- 172.21.15.3:0/3293804631 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4898004c30 con 0x7f48a80b4ff0 2024-03-25T18:26:27.987 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.985+0000 7f489e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:27.987 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.985+0000 7f489e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:27.987 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.985+0000 7f489e7fc640 10 monclient: dump: 2024-03-25T18:26:27.987 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:27.987 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:27.987 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:27.987 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:27.987 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:27.987 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:27.988 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:27.988 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:27.988 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:27.988 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:27.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.985+0000 7f489ffff640 4 set_mon_vals no callback set 2024-03-25T18:26:27.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.986+0000 7f489e7fc640 1 -- 172.21.15.3:0/3293804631 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f489801ac00 con 0x7f48a80b4ff0 2024-03-25T18:26:27.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.987+0000 7f48ade89640 10 monclient: get_auth_request con 0x7f488c06df50 auth_method 0 2024-03-25T18:26:27.989 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.987+0000 7f489e7fc640 10 client.4299.objecter ms_handle_connect 0x7f488c06df50 2024-03-25T18:26:27.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.993+0000 7f489e7fc640 1 -- 172.21.15.3:0/3293804631 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f4898054820 con 0x7f48a80b4ff0 2024-03-25T18:26:27.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.993+0000 7f489e7fc640 10 client.4299.objecter ms_dispatch 0x7f48a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:27.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.993+0000 7f489e7fc640 3 client.4299.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:27.995 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.993+0000 7f489e7fc640 3 client.4299.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:27.995 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.993+0000 7f489e7fc640 20 client.4299.objecter dump_active .. 0 homeless 2024-03-25T18:26:27.995 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.993+0000 7f489e7fc640 1 -- 172.21.15.3:0/3293804631 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4898050160 con 0x7f48a80b4ff0 2024-03-25T18:26:27.995 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.993+0000 7f489e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:27.995 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:27.993+0000 7f489e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:28.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:28.204+0000 7f48b00ed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:28.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:28.204+0000 7f48b00ed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:28.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:28.204+0000 7f48b00ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:26:28.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:28.204+0000 7f48b00ed640 1 -- 172.21.15.3:0/3293804631 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48a80688c0 con 0x7f48a80b4ff0 2024-03-25T18:26:28.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:28.237+0000 7f489e7fc640 1 -- 172.21.15.3:0/3293804631 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f4898001210 con 0x7f48a80b4ff0 2024-03-25T18:26:28.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:28.237+0000 7f489e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:28.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:28.237+0000 7f489e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:28.239 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:26:28.239 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:28.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:28.239+0000 7f48b00ed640 10 librados: watch_flush enter 2024-03-25T18:26:28.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:28.239+0000 7f48b00ed640 10 librados: watch_flush exit 2024-03-25T18:26:28.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:28.239+0000 7f48b00ed640 20 client.4299.objecter shutdown clearing up homeless session... 2024-03-25T18:26:28.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:28.239+0000 7f48b00ed640 10 client.4299.objecter successfully canceled tick 2024-03-25T18:26:28.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:28.239+0000 7f48b00ed640 1 -- 172.21.15.3:0/3293804631 >> v1:172.21.15.3:6800/4195140966 conn(0x7f488c06df50 legacy=0x7f488c070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:28.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:28.239+0000 7f48b00ed640 10 monclient: shutdown 2024-03-25T18:26:28.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:28.239+0000 7f48b00ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:28.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:28.239+0000 7f48b00ed640 1 -- 172.21.15.3:0/3293804631 >> v1:172.21.15.3:6789/0 conn(0x7f48a80b4ff0 legacy=0x7f48a80aee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:28.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:28.239+0000 7f48b00ed640 1 -- 172.21.15.3:0/3293804631 shutdown_connections 2024-03-25T18:26:28.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:28.239+0000 7f48b00ed640 1 -- 172.21.15.3:0/3293804631 >> 172.21.15.3:0/3293804631 conn(0x7f48a80617a0 msgr2=0x7f48a81455d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:28.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:28.240+0000 7f48b00ed640 1 -- 172.21.15.3:0/3293804631 shutdown_connections 2024-03-25T18:26:28.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:28.240+0000 7f48b00ed640 1 -- 172.21.15.3:0/3293804631 wait complete. 2024-03-25T18:26:28.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:28.240+0000 7f48b00ed640 1 librados: shutdown 2024-03-25T18:26:28.251 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:35.059 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:35.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.179+0000 7f6b0d616640 10 monclient: get_monmap_and_config 2024-03-25T18:26:35.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.179+0000 7f6b0d616640 10 monclient: build_initial_monmap 2024-03-25T18:26:35.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.179+0000 7f6b0d616640 10 monclient: monmap: 2024-03-25T18:26:35.181 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:26:35.181 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:35.181 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:26:35.180724+0000 2024-03-25T18:26:35.181 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:26:35.180724+0000 2024-03-25T18:26:35.181 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:35.181 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:35.181 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:26:35.181 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:26:35.182 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:26:35.182 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:35.183 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.180+0000 7f6b0d616640 1 Processor -- start 2024-03-25T18:26:35.183 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.180+0000 7f6b0d616640 1 -- start start 2024-03-25T18:26:35.183 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.180+0000 7f6b0d616640 10 monclient: init 2024-03-25T18:26:35.183 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.181+0000 7f6b0d616640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:35.183 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.181+0000 7f6b0d616640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:26:35.183 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.181+0000 7f6b0d616640 10 monclient(hunting): picked mon.noname-b con 0x7f6b080b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:26:35.183 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.181+0000 7f6b0d616640 10 monclient(hunting): picked mon.noname-a con 0x7f6b080b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:26:35.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.181+0000 7f6b0d616640 10 monclient(hunting): picked mon.noname-c con 0x7f6b0813c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:26:35.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.181+0000 7f6b0d616640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b080ab0e0 con 0x7f6b080b4670 2024-03-25T18:26:35.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.181+0000 7f6b0d616640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b080ab3c0 con 0x7f6b0813c480 2024-03-25T18:26:35.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.181+0000 7f6b0d616640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b080a9f30 con 0x7f6b080b4ff0 2024-03-25T18:26:35.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.181+0000 7f6b0d616640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:35.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.181+0000 7f6b0d616640 10 monclient(hunting): authenticate will time out at 7449.442172s 2024-03-25T18:26:35.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.191+0000 7f6b06ffd640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f6b080b4ff0 0x7f6b0813bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:57732/0 (socket says 172.21.15.3:57732) 2024-03-25T18:26:35.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.191+0000 7f6b06ffd640 1 -- 172.21.15.3:0/1106932968 learned_addr learned my addr 172.21.15.3:0/1106932968 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:26:35.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.200+0000 7f6b067fc640 1 -- 172.21.15.3:0/1106932968 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3605632624 0 0) 0x7f6b080a9f30 con 0x7f6b080b4ff0 2024-03-25T18:26:35.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.200+0000 7f6b067fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:35.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.200+0000 7f6b067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:35.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.200+0000 7f6b067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:35.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.200+0000 7f6b067fc640 10 monclient(hunting): my global_id is 4274 2024-03-25T18:26:35.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.200+0000 7f6b067fc640 1 -- 172.21.15.3:0/1106932968 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6af4003520 con 0x7f6b080b4ff0 2024-03-25T18:26:35.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.209+0000 7f6b067fc640 1 -- 172.21.15.3:0/1106932968 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3982536369 0 0) 0x7f6af4003520 con 0x7f6b080b4ff0 2024-03-25T18:26:35.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.210+0000 7f6b067fc640 1 -- 172.21.15.3:0/1106932968 >> v1:172.21.15.16:6789/0 conn(0x7f6b0813c480 legacy=0x7f6b0814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:35.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.210+0000 7f6b067fc640 1 -- 172.21.15.3:0/1106932968 >> v1:172.21.15.3:6789/0 conn(0x7f6b080b4670 legacy=0x7f6b081425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:35.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.210+0000 7f6b067fc640 10 monclient: _finish_hunting 0 2024-03-25T18:26:35.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.210+0000 7f6b067fc640 1 monclient: found mon.noname-b 2024-03-25T18:26:35.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.210+0000 7f6b067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:35.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.210+0000 7f6b067fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-25T18:26:35.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.210+0000 7f6b067fc640 1 -- 172.21.15.3:0/1106932968 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b080a9b70 con 0x7f6b080b4ff0 2024-03-25T18:26:35.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.210+0000 7f6b067fc640 10 monclient: _finish_auth 0 2024-03-25T18:26:35.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.210+0000 7f6b067fc640 10 monclient: _check_auth_tickets 2024-03-25T18:26:35.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.210+0000 7f6b067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:35.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.210+0000 7f6b067fc640 1 -- 172.21.15.3:0/1106932968 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f6af8001690 con 0x7f6b080b4ff0 2024-03-25T18:26:35.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.210+0000 7f6b067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:35.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.210+0000 7f6b067fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:35.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.210+0000 7f6b067fc640 10 monclient: dump: 2024-03-25T18:26:35.212 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:35.212 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:35.212 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:35.212 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:35.212 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:35.212 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:35.212 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:35.212 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:35.212 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:35.213 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:35.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.210+0000 7f6b0d616640 5 monclient: authenticate success, global_id 4274 2024-03-25T18:26:35.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.210+0000 7f6b0d616640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:35.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.214+0000 7f6b067fc640 1 -- 172.21.15.3:0/1106932968 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6af80017f0 con 0x7f6b080b4ff0 2024-03-25T18:26:35.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.214+0000 7f6b067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:35.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.214+0000 7f6b0d616640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:35.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.214+0000 7f6b067fc640 1 -- 172.21.15.3:0/1106932968 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f6af8001c40 con 0x7f6b080b4ff0 2024-03-25T18:26:35.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.214+0000 7f6b067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:35.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.214+0000 7f6b0d616640 4 set_mon_vals no callback set 2024-03-25T18:26:35.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.214+0000 7f6b067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:35.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.214+0000 7f6b067fc640 10 monclient: dump: 2024-03-25T18:26:35.215 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:35.216 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:35.216 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:35.216 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:35.216 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:35.216 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:35.216 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:35.216 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:35.216 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:35.216 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:35.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.214+0000 7f6b0d616640 10 monclient: shutdown 2024-03-25T18:26:35.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.214+0000 7f6b0d616640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:35.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.214+0000 7f6b0d616640 1 -- 172.21.15.3:0/1106932968 >> v1:172.21.15.3:6790/0 conn(0x7f6b080b4ff0 legacy=0x7f6b0813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:35.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.215+0000 7f6b0d616640 1 -- 172.21.15.3:0/1106932968 shutdown_connections 2024-03-25T18:26:35.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.215+0000 7f6b0d616640 1 -- 172.21.15.3:0/1106932968 >> 172.21.15.3:0/1106932968 conn(0x7f6b080617a0 msgr2=0x7f6b08061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:35.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.215+0000 7f6b0d616640 1 -- 172.21.15.3:0/1106932968 shutdown_connections 2024-03-25T18:26:35.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.215+0000 7f6b0d616640 1 -- 172.21.15.3:0/1106932968 wait complete. 2024-03-25T18:26:35.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.215+0000 7f6b0d616640 10 monclient: build_initial_monmap 2024-03-25T18:26:35.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.215+0000 7f6b0d616640 10 monclient: monmap: 2024-03-25T18:26:35.219 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:26:35.219 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:35.219 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:26:35.219 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:26:35.219 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:35.219 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:26:35.219 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:26:35.219 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:26:35.219 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:26:35.219 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:35.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.216+0000 7f6b0d616640 1 librados: starting msgr at 2024-03-25T18:26:35.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.216+0000 7f6b0d616640 1 librados: starting objecter 2024-03-25T18:26:35.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.216+0000 7f6b0d616640 1 Processor -- start 2024-03-25T18:26:35.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.216+0000 7f6b0d616640 1 -- start start 2024-03-25T18:26:35.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.216+0000 7f6b0d616640 1 librados: setting wanted keys 2024-03-25T18:26:35.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.216+0000 7f6b0d616640 1 librados: calling monclient init 2024-03-25T18:26:35.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.216+0000 7f6b0d616640 10 monclient: init 2024-03-25T18:26:35.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.216+0000 7f6b0d616640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:35.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.216+0000 7f6b0d616640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:35.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.216+0000 7f6b0d616640 10 monclient(hunting): picked mon.noname-a con 0x7f6b080b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:26:35.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.216+0000 7f6b0d616640 10 monclient(hunting): picked mon.noname-c con 0x7f6b080b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:26:35.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.216+0000 7f6b0d616640 10 monclient(hunting): picked mon.noname-b con 0x7f6b0813c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:26:35.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.216+0000 7f6b0d616640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b08144790 con 0x7f6b080b4670 2024-03-25T18:26:35.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.216+0000 7f6b0d616640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b08144a70 con 0x7f6b0813c480 2024-03-25T18:26:35.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.216+0000 7f6b0d616640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b08144d50 con 0x7f6b080b4ff0 2024-03-25T18:26:35.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.216+0000 7f6b0d616640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:35.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.216+0000 7f6b0d616640 10 monclient(hunting): authenticate will time out at 7449.477145s 2024-03-25T18:26:35.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.238+0000 7f6b06ffd640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f6b080b4670 0x7f6b081457c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58890/0 (socket says 172.21.15.3:58890) 2024-03-25T18:26:35.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.238+0000 7f6b06ffd640 1 -- 172.21.15.3:0/2958998794 learned_addr learned my addr 172.21.15.3:0/2958998794 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:26:35.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.238+0000 7f6aff7fe640 10 client.?.objecter ms_handle_connect 0x7f6b080b4670 2024-03-25T18:26:35.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.238+0000 7f6aff7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:35.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.245+0000 7f6aff7fe640 10 client.?.objecter ms_handle_connect 0x7f6b0813c480 2024-03-25T18:26:35.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.245+0000 7f6aff7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:35.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.247+0000 7f6aff7fe640 10 client.?.objecter ms_handle_connect 0x7f6b080b4ff0 2024-03-25T18:26:35.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.247+0000 7f6aff7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:35.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.248+0000 7f6aff7fe640 1 -- 172.21.15.3:0/2958998794 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1618381414 0 0) 0x7f6b08144790 con 0x7f6b080b4670 2024-03-25T18:26:35.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.248+0000 7f6aff7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:35.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.248+0000 7f6aff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:35.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.248+0000 7f6aff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:35.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.248+0000 7f6aff7fe640 10 monclient(hunting): my global_id is 4305 2024-03-25T18:26:35.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.248+0000 7f6aff7fe640 1 -- 172.21.15.3:0/2958998794 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ae8004eb0 con 0x7f6b080b4670 2024-03-25T18:26:35.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.256+0000 7f6aff7fe640 1 -- 172.21.15.3:0/2958998794 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2236413032 0 0) 0x7f6b08144a70 con 0x7f6b0813c480 2024-03-25T18:26:35.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.256+0000 7f6aff7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:35.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.256+0000 7f6aff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:35.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.256+0000 7f6aff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:35.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.256+0000 7f6aff7fe640 10 monclient(hunting): my global_id is 4267 2024-03-25T18:26:35.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.256+0000 7f6aff7fe640 1 -- 172.21.15.3:0/2958998794 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b08144790 con 0x7f6b0813c480 2024-03-25T18:26:35.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.264+0000 7f6aff7fe640 1 -- 172.21.15.3:0/2958998794 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2070850815 0 0) 0x7f6b08144d50 con 0x7f6b080b4ff0 2024-03-25T18:26:35.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.264+0000 7f6aff7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:35.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.264+0000 7f6aff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:35.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.264+0000 7f6aff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:35.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.264+0000 7f6aff7fe640 10 monclient(hunting): my global_id is 4277 2024-03-25T18:26:35.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.264+0000 7f6aff7fe640 1 -- 172.21.15.3:0/2958998794 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b08144a70 con 0x7f6b080b4ff0 2024-03-25T18:26:35.273 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.272+0000 7f6aff7fe640 1 -- 172.21.15.3:0/2958998794 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2433218046 0 0) 0x7f6ae8004eb0 con 0x7f6b080b4670 2024-03-25T18:26:35.273 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.272+0000 7f6aff7fe640 1 -- 172.21.15.3:0/2958998794 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b08144d50 con 0x7f6b080b4670 2024-03-25T18:26:35.273 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.272+0000 7f6aff7fe640 1 -- 172.21.15.3:0/2958998794 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f6af8001b00 con 0x7f6b080b4670 2024-03-25T18:26:35.273 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.272+0000 7f6aff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:35.274 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.272+0000 7f6aff7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:35.274 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.272+0000 7f6aff7fe640 10 monclient(hunting): dump: 2024-03-25T18:26:35.274 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:35.274 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:35.274 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:35.274 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:35.274 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:35.274 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:35.274 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:35.274 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:35.274 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:35.274 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:35.274 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.272+0000 7f6aff7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:35.283 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.281+0000 7f6aff7fe640 1 -- 172.21.15.3:0/2958998794 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 342413154 0 0) 0x7f6b08144790 con 0x7f6b0813c480 2024-03-25T18:26:35.283 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.281+0000 7f6aff7fe640 1 -- 172.21.15.3:0/2958998794 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ae8004eb0 con 0x7f6b0813c480 2024-03-25T18:26:35.283 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.282+0000 7f6aff7fe640 1 -- 172.21.15.3:0/2958998794 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f6b00001b10 con 0x7f6b0813c480 2024-03-25T18:26:35.283 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.282+0000 7f6aff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:35.283 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.282+0000 7f6aff7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:35.283 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.282+0000 7f6aff7fe640 10 monclient(hunting): dump: 2024-03-25T18:26:35.283 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:35.283 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:35.283 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:35.283 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:35.283 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:35.283 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:35.283 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:35.284 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:35.284 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:35.284 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:35.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.291+0000 7f6aff7fe640 1 -- 172.21.15.3:0/2958998794 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1379420066 0 0) 0x7f6b08144d50 con 0x7f6b080b4670 2024-03-25T18:26:35.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.291+0000 7f6aff7fe640 1 -- 172.21.15.3:0/2958998794 >> v1:172.21.15.3:6790/0 conn(0x7f6b080b4ff0 legacy=0x7f6b08145ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:35.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.291+0000 7f6aff7fe640 1 -- 172.21.15.3:0/2958998794 >> v1:172.21.15.16:6789/0 conn(0x7f6b0813c480 legacy=0x7f6b081e2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:35.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.291+0000 7f6aff7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:26:35.294 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.291+0000 7f6aff7fe640 1 monclient: found mon.a 2024-03-25T18:26:35.294 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.291+0000 7f6aff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:35.294 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.291+0000 7f6aff7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:26:35.294 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.291+0000 7f6aff7fe640 1 -- 172.21.15.3:0/2958998794 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b08145030 con 0x7f6b080b4670 2024-03-25T18:26:35.294 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.291+0000 7f6aff7fe640 10 monclient: _finish_auth 0 2024-03-25T18:26:35.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.291+0000 7f6aff7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:26:35.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.292+0000 7f6aff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:35.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.292+0000 7f6b0d616640 5 monclient: authenticate success, global_id 4305 2024-03-25T18:26:35.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.292+0000 7f6b0d616640 10 monclient: _renew_subs 2024-03-25T18:26:35.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.292+0000 7f6b0d616640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:26:35.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.292+0000 7f6b0d616640 1 -- 172.21.15.3:0/2958998794 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b081435a0 con 0x7f6b080b4670 2024-03-25T18:26:35.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.292+0000 7f6b0d616640 10 client.4305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:35.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.292+0000 7f6b0d616640 10 monclient: _renew_subs 2024-03-25T18:26:35.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.292+0000 7f6b0d616640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:26:35.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.292+0000 7f6b0d616640 1 -- 172.21.15.3:0/2958998794 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b0813dbe0 con 0x7f6b080b4670 2024-03-25T18:26:35.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.292+0000 7f6b0d616640 1 librados: init done 2024-03-25T18:26:35.296 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.293+0000 7f6b0d616640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:35.296 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.293+0000 7f6b0d616640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:35.296 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.293+0000 7f6b0d616640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:26:35.296 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.293+0000 7f6b0d616640 1 -- 172.21.15.3:0/2958998794 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b080688c0 con 0x7f6b080b4670 2024-03-25T18:26:35.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.514+0000 7f6aff7fe640 1 -- 172.21.15.3:0/2958998794 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6af8001d20 con 0x7f6b080b4670 2024-03-25T18:26:35.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.514+0000 7f6aff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:35.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.514+0000 7f6aff7fe640 1 -- 172.21.15.3:0/2958998794 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f6af8004d90 con 0x7f6b080b4670 2024-03-25T18:26:35.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.514+0000 7f6aff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:35.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.514+0000 7f6aff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:35.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.514+0000 7f6aff7fe640 10 monclient: dump: 2024-03-25T18:26:35.516 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:35.516 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:35.516 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:35.516 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:35.516 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:35.516 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:35.516 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:35.516 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:35.516 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:35.516 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:35.516 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.514+0000 7f6b05ffb640 4 set_mon_vals no callback set 2024-03-25T18:26:35.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.542+0000 7f6aff7fe640 1 -- 172.21.15.3:0/2958998794 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f6af801ad60 con 0x7f6b080b4670 2024-03-25T18:26:35.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.542+0000 7f6afffff640 10 monclient: get_auth_request con 0x7f6ae806df20 auth_method 0 2024-03-25T18:26:35.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.542+0000 7f6aff7fe640 10 client.4305.objecter ms_handle_connect 0x7f6ae806df20 2024-03-25T18:26:35.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.623+0000 7f6aff7fe640 1 -- 172.21.15.3:0/2958998794 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f6af8054910 con 0x7f6b080b4670 2024-03-25T18:26:35.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.623+0000 7f6aff7fe640 10 client.4305.objecter ms_dispatch 0x7f6b08000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:35.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.623+0000 7f6aff7fe640 3 client.4305.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:35.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.623+0000 7f6aff7fe640 3 client.4305.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:35.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.623+0000 7f6aff7fe640 20 client.4305.objecter dump_active .. 0 homeless 2024-03-25T18:26:35.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.623+0000 7f6aff7fe640 1 -- 172.21.15.3:0/2958998794 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6af8050250 con 0x7f6b080b4670 2024-03-25T18:26:35.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.623+0000 7f6aff7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:35.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.623+0000 7f6aff7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:35.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.843+0000 7f6b0d616640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:35.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.844+0000 7f6b0d616640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:35.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.844+0000 7f6b0d616640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:26:35.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.844+0000 7f6b0d616640 1 -- 172.21.15.3:0/2958998794 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b0806dbd0 con 0x7f6b080b4670 2024-03-25T18:26:35.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.887+0000 7f6aff7fe640 1 -- 172.21.15.3:0/2958998794 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f6af8052350 con 0x7f6b080b4670 2024-03-25T18:26:35.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.887+0000 7f6aff7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:35.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.887+0000 7f6aff7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:35.889 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:26:35.889 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:35.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.889+0000 7f6b0d616640 10 librados: watch_flush enter 2024-03-25T18:26:35.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.889+0000 7f6b0d616640 10 librados: watch_flush exit 2024-03-25T18:26:35.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.890+0000 7f6b0d616640 20 client.4305.objecter shutdown clearing up homeless session... 2024-03-25T18:26:35.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.890+0000 7f6b0d616640 10 client.4305.objecter successfully canceled tick 2024-03-25T18:26:35.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.890+0000 7f6b0d616640 1 -- 172.21.15.3:0/2958998794 >> v1:172.21.15.3:6800/4195140966 conn(0x7f6ae806df20 legacy=0x7f6ae80703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:35.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.890+0000 7f6b0d616640 10 monclient: shutdown 2024-03-25T18:26:35.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.890+0000 7f6b0d616640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:35.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.890+0000 7f6b0d616640 1 -- 172.21.15.3:0/2958998794 >> v1:172.21.15.3:6789/0 conn(0x7f6b080b4670 legacy=0x7f6b081457c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:35.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.890+0000 7f6b0d616640 1 -- 172.21.15.3:0/2958998794 shutdown_connections 2024-03-25T18:26:35.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.890+0000 7f6b0d616640 1 -- 172.21.15.3:0/2958998794 >> 172.21.15.3:0/2958998794 conn(0x7f6b080617a0 msgr2=0x7f6b08140760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:35.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.890+0000 7f6b0d616640 1 -- 172.21.15.3:0/2958998794 shutdown_connections 2024-03-25T18:26:35.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.890+0000 7f6b0d616640 1 -- 172.21.15.3:0/2958998794 wait complete. 2024-03-25T18:26:35.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:35.890+0000 7f6b0d616640 1 librados: shutdown 2024-03-25T18:26:35.904 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:40.799 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:01:54.528 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:40.817 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:01:54.547 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:40.844 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:01:54.573 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:40.846 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:01:54.575 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:40.863 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:01:54.555 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:40.867 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:01:54.596 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:40.872 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:01:54.564 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:40.889 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:01:54.580 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:40.894 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:01:54.623 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:42.711 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:42.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.826+0000 7fc016493640 10 monclient: get_monmap_and_config 2024-03-25T18:26:42.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.826+0000 7fc016493640 10 monclient: build_initial_monmap 2024-03-25T18:26:42.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.826+0000 7fc016493640 10 monclient: monmap: 2024-03-25T18:26:42.827 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:26:42.828 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:42.828 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:26:42.827498+0000 2024-03-25T18:26:42.828 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:26:42.827498+0000 2024-03-25T18:26:42.828 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:42.828 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:42.828 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:26:42.828 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:26:42.828 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:26:42.828 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:42.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.827+0000 7fc016493640 1 Processor -- start 2024-03-25T18:26:42.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.827+0000 7fc016493640 1 -- start start 2024-03-25T18:26:42.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.827+0000 7fc016493640 10 monclient: init 2024-03-25T18:26:42.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.828+0000 7fc016493640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:42.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.828+0000 7fc016493640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:26:42.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.828+0000 7fc016493640 10 monclient(hunting): picked mon.noname-b con 0x7fc0100b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:26:42.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.828+0000 7fc016493640 10 monclient(hunting): picked mon.noname-c con 0x7fc0100b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:26:42.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.828+0000 7fc016493640 10 monclient(hunting): picked mon.noname-a con 0x7fc01013c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:26:42.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.828+0000 7fc016493640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0100ab0e0 con 0x7fc01013c480 2024-03-25T18:26:42.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.828+0000 7fc016493640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0100ab3c0 con 0x7fc0100b4670 2024-03-25T18:26:42.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.828+0000 7fc016493640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0100a9f30 con 0x7fc0100b4ff0 2024-03-25T18:26:42.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.828+0000 7fc016493640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:42.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.828+0000 7fc016493640 10 monclient(hunting): authenticate will time out at 7457.088543s 2024-03-25T18:26:42.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.834+0000 7fc00ffff640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fc0100b4ff0 0x7fc01013bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:54096/0 (socket says 172.21.15.3:54096) 2024-03-25T18:26:42.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.834+0000 7fc00ffff640 1 -- 172.21.15.3:0/3649215251 learned_addr learned my addr 172.21.15.3:0/3649215251 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:26:42.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.843+0000 7fc00effd640 1 -- 172.21.15.3:0/3649215251 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 498707148 0 0) 0x7fc0100a9f30 con 0x7fc0100b4ff0 2024-03-25T18:26:42.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.843+0000 7fc00effd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:42.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.843+0000 7fc00effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:42.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.843+0000 7fc00effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:42.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.843+0000 7fc00effd640 10 monclient(hunting): my global_id is 4280 2024-03-25T18:26:42.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.843+0000 7fc00effd640 1 -- 172.21.15.3:0/3649215251 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbff8003520 con 0x7fc0100b4ff0 2024-03-25T18:26:42.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.850+0000 7fc00effd640 1 -- 172.21.15.3:0/3649215251 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 215477902 0 0) 0x7fc0100ab3c0 con 0x7fc0100b4670 2024-03-25T18:26:42.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.850+0000 7fc00effd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:42.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.850+0000 7fc00effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:42.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.850+0000 7fc00effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:42.852 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.850+0000 7fc00effd640 10 monclient(hunting): my global_id is 4270 2024-03-25T18:26:42.852 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.850+0000 7fc00effd640 1 -- 172.21.15.3:0/3649215251 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0100a9f30 con 0x7fc0100b4670 2024-03-25T18:26:42.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.855+0000 7fc00effd640 1 -- 172.21.15.3:0/3649215251 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3287992906 0 0) 0x7fc0100a9f30 con 0x7fc0100b4670 2024-03-25T18:26:42.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.855+0000 7fc00effd640 1 -- 172.21.15.3:0/3649215251 >> v1:172.21.15.3:6790/0 conn(0x7fc0100b4ff0 legacy=0x7fc01013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:42.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.855+0000 7fc00effd640 1 -- 172.21.15.3:0/3649215251 >> v1:172.21.15.3:6789/0 conn(0x7fc01013c480 legacy=0x7fc01014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:42.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.855+0000 7fc00effd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:42.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.855+0000 7fc00effd640 1 monclient: found mon.noname-c 2024-03-25T18:26:42.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.855+0000 7fc00effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:42.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.855+0000 7fc00effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:26:42.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.855+0000 7fc00effd640 1 -- 172.21.15.3:0/3649215251 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0100a9b70 con 0x7fc0100b4670 2024-03-25T18:26:42.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.855+0000 7fc00effd640 10 monclient: _finish_auth 0 2024-03-25T18:26:42.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.855+0000 7fc00effd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:42.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.855+0000 7fc00effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:42.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.855+0000 7fc00effd640 1 -- 172.21.15.3:0/3649215251 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc004001670 con 0x7fc0100b4670 2024-03-25T18:26:42.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.855+0000 7fc00effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:42.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.855+0000 7fc00effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:42.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.855+0000 7fc00effd640 10 monclient: dump: 2024-03-25T18:26:42.857 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:42.857 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:42.857 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:42.857 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:42.858 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:42.858 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:42.858 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:42.858 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:42.858 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:42.858 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:42.858 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.855+0000 7fc016493640 5 monclient: authenticate success, global_id 4270 2024-03-25T18:26:42.858 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.855+0000 7fc016493640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:42.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.870+0000 7fc00effd640 1 -- 172.21.15.3:0/3649215251 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0040017d0 con 0x7fc0100b4670 2024-03-25T18:26:42.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.870+0000 7fc00effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:42.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.870+0000 7fc00effd640 1 -- 172.21.15.3:0/3649215251 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc004001ba0 con 0x7fc0100b4670 2024-03-25T18:26:42.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.870+0000 7fc00effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:42.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.870+0000 7fc00effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:42.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.870+0000 7fc00effd640 10 monclient: dump: 2024-03-25T18:26:42.872 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:42.872 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:42.872 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:42.872 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:42.872 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:42.872 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:42.872 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:42.872 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:42.873 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:42.873 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:42.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.870+0000 7fc016493640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:42.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.870+0000 7fc016493640 4 set_mon_vals no callback set 2024-03-25T18:26:42.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.871+0000 7fc016493640 10 monclient: shutdown 2024-03-25T18:26:42.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.871+0000 7fc016493640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:42.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.871+0000 7fc016493640 1 -- 172.21.15.3:0/3649215251 >> v1:172.21.15.16:6789/0 conn(0x7fc0100b4670 legacy=0x7fc0101425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:42.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.871+0000 7fc016493640 1 -- 172.21.15.3:0/3649215251 shutdown_connections 2024-03-25T18:26:42.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.871+0000 7fc016493640 1 -- 172.21.15.3:0/3649215251 >> 172.21.15.3:0/3649215251 conn(0x7fc0100617a0 msgr2=0x7fc010061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:42.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.871+0000 7fc016493640 1 -- 172.21.15.3:0/3649215251 shutdown_connections 2024-03-25T18:26:42.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.871+0000 7fc016493640 1 -- 172.21.15.3:0/3649215251 wait complete. 2024-03-25T18:26:42.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.872+0000 7fc016493640 10 monclient: build_initial_monmap 2024-03-25T18:26:42.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.872+0000 7fc016493640 10 monclient: monmap: 2024-03-25T18:26:42.875 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:26:42.875 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:42.875 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:26:42.876 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:26:42.876 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:42.876 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:26:42.876 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:26:42.876 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:26:42.876 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:26:42.876 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:42.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.872+0000 7fc016493640 1 librados: starting msgr at 2024-03-25T18:26:42.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.872+0000 7fc016493640 1 librados: starting objecter 2024-03-25T18:26:42.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.872+0000 7fc016493640 1 Processor -- start 2024-03-25T18:26:42.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.872+0000 7fc016493640 1 -- start start 2024-03-25T18:26:42.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.872+0000 7fc016493640 1 librados: setting wanted keys 2024-03-25T18:26:42.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.872+0000 7fc016493640 1 librados: calling monclient init 2024-03-25T18:26:42.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.873+0000 7fc016493640 10 monclient: init 2024-03-25T18:26:42.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.873+0000 7fc016493640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:42.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.873+0000 7fc016493640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:26:42.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.874+0000 7fc016493640 10 monclient(hunting): picked mon.noname-c con 0x7fc0100b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:26:42.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.874+0000 7fc016493640 10 monclient(hunting): picked mon.noname-b con 0x7fc0100b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:26:42.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.874+0000 7fc016493640 10 monclient(hunting): picked mon.noname-a con 0x7fc01013c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:26:42.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.874+0000 7fc016493640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc01013c8f0 con 0x7fc01013c480 2024-03-25T18:26:42.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.874+0000 7fc016493640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc01013c2b0 con 0x7fc0100b4ff0 2024-03-25T18:26:42.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.874+0000 7fc016493640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc010130ad0 con 0x7fc0100b4670 2024-03-25T18:26:42.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.874+0000 7fc016493640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:42.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.874+0000 7fc016493640 10 monclient(hunting): authenticate will time out at 7457.134491s 2024-03-25T18:26:42.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.875+0000 7fc00f7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc0100b4ff0 0x7fc01012fcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:35744/0 (socket says 172.21.15.3:35744) 2024-03-25T18:26:42.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.875+0000 7fc00f7fe640 1 -- 172.21.15.3:0/861568074 learned_addr learned my addr 172.21.15.3:0/861568074 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:26:42.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.876+0000 7fc00cff9640 10 client.?.objecter ms_handle_connect 0x7fc0100b4ff0 2024-03-25T18:26:42.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.876+0000 7fc00cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:42.881 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.880+0000 7fc00cff9640 10 client.?.objecter ms_handle_connect 0x7fc0100b4670 2024-03-25T18:26:42.881 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.880+0000 7fc00cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:42.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.883+0000 7fc00cff9640 1 -- 172.21.15.3:0/861568074 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 436986059 0 0) 0x7fc01013c2b0 con 0x7fc0100b4ff0 2024-03-25T18:26:42.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.883+0000 7fc00cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:42.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.883+0000 7fc00cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:42.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.883+0000 7fc00cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:42.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.883+0000 7fc00cff9640 10 monclient(hunting): my global_id is 4273 2024-03-25T18:26:42.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.883+0000 7fc00cff9640 1 -- 172.21.15.3:0/861568074 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbffc004eb0 con 0x7fc0100b4ff0 2024-03-25T18:26:42.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.891+0000 7fc00cff9640 10 client.?.objecter ms_handle_connect 0x7fc01013c480 2024-03-25T18:26:42.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.891+0000 7fc00cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:42.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.905+0000 7fc00cff9640 1 -- 172.21.15.3:0/861568074 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1315554392 0 0) 0x7fc010130ad0 con 0x7fc0100b4670 2024-03-25T18:26:42.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.905+0000 7fc00cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:42.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.905+0000 7fc00cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:42.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.905+0000 7fc00cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:42.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.905+0000 7fc00cff9640 10 monclient(hunting): my global_id is 4283 2024-03-25T18:26:42.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.905+0000 7fc00cff9640 1 -- 172.21.15.3:0/861568074 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc01013c2b0 con 0x7fc0100b4670 2024-03-25T18:26:42.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.906+0000 7fc00cff9640 1 -- 172.21.15.3:0/861568074 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4177059052 0 0) 0x7fc01013c8f0 con 0x7fc01013c480 2024-03-25T18:26:42.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.906+0000 7fc00cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:42.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.906+0000 7fc00cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:42.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.906+0000 7fc00cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:42.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.906+0000 7fc00cff9640 10 monclient(hunting): my global_id is 4311 2024-03-25T18:26:42.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.906+0000 7fc00cff9640 1 -- 172.21.15.3:0/861568074 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc010130ad0 con 0x7fc01013c480 2024-03-25T18:26:42.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.915+0000 7fc00cff9640 1 -- 172.21.15.3:0/861568074 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3652186803 0 0) 0x7fbffc004eb0 con 0x7fc0100b4ff0 2024-03-25T18:26:42.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.915+0000 7fc00cff9640 1 -- 172.21.15.3:0/861568074 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc01013c8f0 con 0x7fc0100b4ff0 2024-03-25T18:26:42.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.915+0000 7fc00cff9640 1 -- 172.21.15.3:0/861568074 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc004001be0 con 0x7fc0100b4ff0 2024-03-25T18:26:42.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.915+0000 7fc00cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:42.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.915+0000 7fc00cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:42.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.915+0000 7fc00cff9640 10 monclient(hunting): dump: 2024-03-25T18:26:42.917 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:42.917 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:42.917 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:42.917 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:42.917 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:42.917 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:42.917 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:42.917 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:42.917 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:42.918 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:42.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.915+0000 7fc00cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:42.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.919+0000 7fc00cff9640 1 -- 172.21.15.3:0/861568074 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 613019608 0 0) 0x7fc010130ad0 con 0x7fc01013c480 2024-03-25T18:26:42.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.919+0000 7fc00cff9640 1 -- 172.21.15.3:0/861568074 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbffc004eb0 con 0x7fc01013c480 2024-03-25T18:26:42.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.919+0000 7fc00cff9640 1 -- 172.21.15.3:0/861568074 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc008001a90 con 0x7fc01013c480 2024-03-25T18:26:42.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.919+0000 7fc00cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:42.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.919+0000 7fc00cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:42.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.919+0000 7fc00cff9640 10 monclient(hunting): dump: 2024-03-25T18:26:42.920 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:42.920 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:42.921 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:42.921 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:42.921 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:42.921 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:42.921 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:42.921 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:42.921 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:42.921 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:42.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.922+0000 7fc00cff9640 1 -- 172.21.15.3:0/861568074 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2207866621 0 0) 0x7fc01013c2b0 con 0x7fc0100b4670 2024-03-25T18:26:42.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.922+0000 7fc00cff9640 1 -- 172.21.15.3:0/861568074 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc010130ad0 con 0x7fc0100b4670 2024-03-25T18:26:42.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.922+0000 7fc00cff9640 1 -- 172.21.15.3:0/861568074 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc000001950 con 0x7fc0100b4670 2024-03-25T18:26:42.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.922+0000 7fc00cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:42.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.922+0000 7fc00cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:42.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.922+0000 7fc00cff9640 10 monclient(hunting): dump: 2024-03-25T18:26:42.924 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:42.924 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:42.924 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:42.924 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:42.924 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:42.924 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:42.924 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:42.924 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:42.924 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:42.924 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:42.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.936+0000 7fc00cff9640 1 -- 172.21.15.3:0/861568074 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 343915831 0 0) 0x7fc010130ad0 con 0x7fc0100b4670 2024-03-25T18:26:42.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.936+0000 7fc00cff9640 1 -- 172.21.15.3:0/861568074 >> v1:172.21.15.16:6789/0 conn(0x7fc0100b4ff0 legacy=0x7fc01012fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:42.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.936+0000 7fc00cff9640 1 -- 172.21.15.3:0/861568074 >> v1:172.21.15.3:6789/0 conn(0x7fc01013c480 legacy=0x7fc0101303c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:42.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.936+0000 7fc00cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:26:42.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.936+0000 7fc00cff9640 1 monclient: found mon.c 2024-03-25T18:26:42.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.936+0000 7fc00cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:42.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.936+0000 7fc00cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:26:42.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.936+0000 7fc00cff9640 1 -- 172.21.15.3:0/861568074 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc01013e8f0 con 0x7fc0100b4670 2024-03-25T18:26:42.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.936+0000 7fc00cff9640 10 monclient: _finish_auth 0 2024-03-25T18:26:42.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.936+0000 7fc00cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:26:42.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.936+0000 7fc00cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:42.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.936+0000 7fc016493640 5 monclient: authenticate success, global_id 4283 2024-03-25T18:26:42.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.936+0000 7fc016493640 10 monclient: _renew_subs 2024-03-25T18:26:42.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.936+0000 7fc016493640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:26:42.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.936+0000 7fc016493640 1 -- 172.21.15.3:0/861568074 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc01013d790 con 0x7fc0100b4670 2024-03-25T18:26:42.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.936+0000 7fc016493640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:42.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.936+0000 7fc016493640 10 monclient: _renew_subs 2024-03-25T18:26:42.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.936+0000 7fc016493640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:26:42.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.936+0000 7fc016493640 1 -- 172.21.15.3:0/861568074 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc01013db20 con 0x7fc0100b4670 2024-03-25T18:26:42.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.936+0000 7fc016493640 1 librados: init done 2024-03-25T18:26:42.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.937+0000 7fc016493640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:42.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.937+0000 7fc016493640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:42.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.937+0000 7fc016493640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:26:42.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:42.937+0000 7fc016493640 1 -- 172.21.15.3:0/861568074 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc01013cc60 con 0x7fc0100b4670 2024-03-25T18:26:43.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.240+0000 7fc00cff9640 1 -- 172.21.15.3:0/861568074 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc000001d60 con 0x7fc0100b4670 2024-03-25T18:26:43.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.240+0000 7fc00cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:43.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.240+0000 7fc00cff9640 1 -- 172.21.15.3:0/861568074 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc000004980 con 0x7fc0100b4670 2024-03-25T18:26:43.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.240+0000 7fc00cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:43.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.240+0000 7fc00cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:43.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.240+0000 7fc00cff9640 10 monclient: dump: 2024-03-25T18:26:43.242 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:43.242 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:43.242 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:43.242 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:43.242 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:43.242 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:43.242 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:43.242 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:43.243 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:43.243 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:43.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.240+0000 7fc00e7fc640 4 set_mon_vals no callback set 2024-03-25T18:26:43.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.241+0000 7fc00cff9640 1 -- 172.21.15.3:0/861568074 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7fc00001a950 con 0x7fc0100b4670 2024-03-25T18:26:43.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.241+0000 7fc00f7fe640 10 monclient: get_auth_request con 0x7fbffc06e060 auth_method 0 2024-03-25T18:26:43.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.242+0000 7fc00cff9640 10 client.4283.objecter ms_handle_connect 0x7fbffc06e060 2024-03-25T18:26:43.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.250+0000 7fc00cff9640 1 -- 172.21.15.3:0/861568074 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7fc000054710 con 0x7fc0100b4670 2024-03-25T18:26:43.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.250+0000 7fc00cff9640 10 client.4283.objecter ms_dispatch 0x7fc010000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:43.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.250+0000 7fc00cff9640 3 client.4283.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:43.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.250+0000 7fc00cff9640 3 client.4283.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:43.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.250+0000 7fc00cff9640 20 client.4283.objecter dump_active .. 0 homeless 2024-03-25T18:26:43.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.250+0000 7fc00cff9640 1 -- 172.21.15.3:0/861568074 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc000050050 con 0x7fc0100b4670 2024-03-25T18:26:43.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.250+0000 7fc00cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:43.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.250+0000 7fc00cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:43.510 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.509+0000 7fc016493640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:43.510 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.509+0000 7fc016493640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:43.510 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.509+0000 7fc016493640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:26:43.510 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.509+0000 7fc016493640 1 -- 172.21.15.3:0/861568074 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc010146790 con 0x7fc0100b4670 2024-03-25T18:26:43.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.537+0000 7fc00cff9640 1 -- 172.21.15.3:0/861568074 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7fc000052150 con 0x7fc0100b4670 2024-03-25T18:26:43.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.537+0000 7fc00cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:43.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.537+0000 7fc00cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:43.538 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:26:43.539 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:43.540 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.539+0000 7fc016493640 10 librados: watch_flush enter 2024-03-25T18:26:43.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.539+0000 7fc016493640 10 librados: watch_flush exit 2024-03-25T18:26:43.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.539+0000 7fc016493640 20 client.4283.objecter shutdown clearing up homeless session... 2024-03-25T18:26:43.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.539+0000 7fc016493640 10 client.4283.objecter successfully canceled tick 2024-03-25T18:26:43.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.539+0000 7fc016493640 1 -- 172.21.15.3:0/861568074 >> v1:172.21.15.3:6800/4195140966 conn(0x7fbffc06e060 legacy=0x7fbffc070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:43.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.539+0000 7fc016493640 10 monclient: shutdown 2024-03-25T18:26:43.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.539+0000 7fc016493640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:43.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.539+0000 7fc016493640 1 -- 172.21.15.3:0/861568074 >> v1:172.21.15.3:6790/0 conn(0x7fc0100b4670 legacy=0x7fc01013e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:43.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.539+0000 7fc016493640 1 -- 172.21.15.3:0/861568074 shutdown_connections 2024-03-25T18:26:43.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.539+0000 7fc016493640 1 -- 172.21.15.3:0/861568074 >> 172.21.15.3:0/861568074 conn(0x7fc0100617a0 msgr2=0x7fc0100b2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:43.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.539+0000 7fc016493640 1 -- 172.21.15.3:0/861568074 shutdown_connections 2024-03-25T18:26:43.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.539+0000 7fc016493640 1 -- 172.21.15.3:0/861568074 wait complete. 2024-03-25T18:26:43.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:43.539+0000 7fc016493640 1 librados: shutdown 2024-03-25T18:26:43.553 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:50.359 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:50.476 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.475+0000 7f95874f1640 10 monclient: get_monmap_and_config 2024-03-25T18:26:50.476 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.475+0000 7f95874f1640 10 monclient: build_initial_monmap 2024-03-25T18:26:50.476 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.475+0000 7f95874f1640 10 monclient: monmap: 2024-03-25T18:26:50.476 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:26:50.476 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:50.476 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:26:50.476301+0000 2024-03-25T18:26:50.476 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:26:50.476301+0000 2024-03-25T18:26:50.476 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:50.477 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:50.477 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:26:50.477 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:26:50.477 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:26:50.477 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:50.478 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.476+0000 7f95874f1640 1 Processor -- start 2024-03-25T18:26:50.478 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.476+0000 7f95874f1640 1 -- start start 2024-03-25T18:26:50.478 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.476+0000 7f95874f1640 10 monclient: init 2024-03-25T18:26:50.478 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.476+0000 7f95874f1640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:50.478 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.477+0000 7f95874f1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:50.478 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.477+0000 7f95874f1640 10 monclient(hunting): picked mon.noname-a con 0x7f95800b4e70 addr v1:172.21.15.3:6789/0 2024-03-25T18:26:50.478 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.477+0000 7f95874f1640 10 monclient(hunting): picked mon.noname-b con 0x7f95800b44b0 addr v1:172.21.15.3:6790/0 2024-03-25T18:26:50.478 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.477+0000 7f95874f1640 10 monclient(hunting): picked mon.noname-c con 0x7f95800b1b00 addr v1:172.21.15.16:6789/0 2024-03-25T18:26:50.479 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.477+0000 7f95874f1640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95800a96f0 con 0x7f95800b4e70 2024-03-25T18:26:50.479 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.477+0000 7f95874f1640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95800ab0e0 con 0x7f95800b1b00 2024-03-25T18:26:50.479 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.477+0000 7f95874f1640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9580145f70 con 0x7f95800b44b0 2024-03-25T18:26:50.479 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.477+0000 7f95874f1640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:50.479 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.477+0000 7f95874f1640 10 monclient(hunting): authenticate will time out at 7464.737472s 2024-03-25T18:26:50.481 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.480+0000 7f9584a8c640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f95800b44b0 0x7f9580145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:39592/0 (socket says 172.21.15.3:39592) 2024-03-25T18:26:50.481 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.480+0000 7f9584a8c640 1 -- 172.21.15.3:0/3733476950 learned_addr learned my addr 172.21.15.3:0/3733476950 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:26:50.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.498+0000 7f9577fff640 1 -- 172.21.15.3:0/3733476950 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 690662224 0 0) 0x7f9580145f70 con 0x7f95800b44b0 2024-03-25T18:26:50.500 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.498+0000 7f9577fff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:50.500 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.498+0000 7f9577fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:50.500 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.498+0000 7f9577fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:50.500 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.498+0000 7f9577fff640 10 monclient(hunting): my global_id is 4286 2024-03-25T18:26:50.500 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.498+0000 7f9577fff640 1 -- 172.21.15.3:0/3733476950 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f956c003520 con 0x7f95800b44b0 2024-03-25T18:26:50.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.505+0000 7f9577fff640 1 -- 172.21.15.3:0/3733476950 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1257329668 0 0) 0x7f95800ab0e0 con 0x7f95800b1b00 2024-03-25T18:26:50.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.505+0000 7f9577fff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:50.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.505+0000 7f9577fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:50.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.505+0000 7f9577fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:50.507 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.505+0000 7f9577fff640 10 monclient(hunting): my global_id is 4276 2024-03-25T18:26:50.507 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.505+0000 7f9577fff640 1 -- 172.21.15.3:0/3733476950 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9580145f70 con 0x7f95800b1b00 2024-03-25T18:26:50.510 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.509+0000 7f9577fff640 1 -- 172.21.15.3:0/3733476950 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1061338979 0 0) 0x7f95800a96f0 con 0x7f95800b4e70 2024-03-25T18:26:50.510 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.509+0000 7f9577fff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:50.510 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.509+0000 7f9577fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:50.510 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.509+0000 7f9577fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:50.510 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.509+0000 7f9577fff640 10 monclient(hunting): my global_id is 4314 2024-03-25T18:26:50.511 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.509+0000 7f9577fff640 1 -- 172.21.15.3:0/3733476950 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95800ab0e0 con 0x7f95800b4e70 2024-03-25T18:26:50.517 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.515+0000 7f9577fff640 1 -- 172.21.15.3:0/3733476950 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1998711505 0 0) 0x7f95800ab0e0 con 0x7f95800b4e70 2024-03-25T18:26:50.517 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.515+0000 7f9577fff640 1 -- 172.21.15.3:0/3733476950 >> v1:172.21.15.3:6790/0 conn(0x7f95800b44b0 legacy=0x7f9580145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:50.517 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.515+0000 7f9577fff640 1 -- 172.21.15.3:0/3733476950 >> v1:172.21.15.16:6789/0 conn(0x7f95800b1b00 legacy=0x7f958014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:50.517 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.515+0000 7f9577fff640 10 monclient: _finish_hunting 0 2024-03-25T18:26:50.517 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.515+0000 7f9577fff640 1 monclient: found mon.noname-a 2024-03-25T18:26:50.517 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.515+0000 7f9577fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:50.517 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.515+0000 7f9577fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:26:50.517 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.515+0000 7f9577fff640 1 -- 172.21.15.3:0/3733476950 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95800a9b80 con 0x7f95800b4e70 2024-03-25T18:26:50.517 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.515+0000 7f9577fff640 10 monclient: _finish_auth 0 2024-03-25T18:26:50.517 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.515+0000 7f9577fff640 10 monclient: _check_auth_tickets 2024-03-25T18:26:50.517 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.515+0000 7f9577fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:50.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.515+0000 7f9577fff640 1 -- 172.21.15.3:0/3733476950 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f957c0015b0 con 0x7f95800b4e70 2024-03-25T18:26:50.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.515+0000 7f9577fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:50.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.515+0000 7f9577fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:50.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.515+0000 7f9577fff640 10 monclient: dump: 2024-03-25T18:26:50.518 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:50.518 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:50.518 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:50.518 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:50.518 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:50.518 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:50.518 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:50.518 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:50.519 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:50.519 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:50.519 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.515+0000 7f95874f1640 5 monclient: authenticate success, global_id 4314 2024-03-25T18:26:50.519 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.515+0000 7f95874f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:50.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.532+0000 7f9577fff640 1 -- 172.21.15.3:0/3733476950 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f957c001790 con 0x7f95800b4e70 2024-03-25T18:26:50.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.532+0000 7f9577fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:50.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.532+0000 7f9577fff640 1 -- 172.21.15.3:0/3733476950 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f957c001ba0 con 0x7f95800b4e70 2024-03-25T18:26:50.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.532+0000 7f9577fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:50.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.532+0000 7f9577fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:50.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.532+0000 7f9577fff640 10 monclient: dump: 2024-03-25T18:26:50.534 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:50.534 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:50.534 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:50.534 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:50.534 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:50.534 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:50.534 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:50.534 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:50.534 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:50.534 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:50.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.532+0000 7f95874f1640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:50.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.532+0000 7f95874f1640 4 set_mon_vals no callback set 2024-03-25T18:26:50.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.533+0000 7f95874f1640 10 monclient: shutdown 2024-03-25T18:26:50.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.533+0000 7f95874f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:50.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.533+0000 7f95874f1640 1 -- 172.21.15.3:0/3733476950 >> v1:172.21.15.3:6789/0 conn(0x7f95800b4e70 legacy=0x7f958013f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:50.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.533+0000 7f95874f1640 1 -- 172.21.15.3:0/3733476950 shutdown_connections 2024-03-25T18:26:50.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.533+0000 7f95874f1640 1 -- 172.21.15.3:0/3733476950 >> 172.21.15.3:0/3733476950 conn(0x7f95800617a0 msgr2=0x7f9580061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:50.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.533+0000 7f95874f1640 1 -- 172.21.15.3:0/3733476950 shutdown_connections 2024-03-25T18:26:50.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.533+0000 7f95874f1640 1 -- 172.21.15.3:0/3733476950 wait complete. 2024-03-25T18:26:50.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.534+0000 7f95874f1640 10 monclient: build_initial_monmap 2024-03-25T18:26:50.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.534+0000 7f95874f1640 10 monclient: monmap: 2024-03-25T18:26:50.537 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:26:50.537 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:50.537 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:26:50.537 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:26:50.537 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:50.537 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:26:50.537 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:26:50.537 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:26:50.537 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:26:50.537 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:50.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.535+0000 7f95874f1640 1 librados: starting msgr at 2024-03-25T18:26:50.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.535+0000 7f95874f1640 1 librados: starting objecter 2024-03-25T18:26:50.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.535+0000 7f95874f1640 1 Processor -- start 2024-03-25T18:26:50.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.535+0000 7f95874f1640 1 -- start start 2024-03-25T18:26:50.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.535+0000 7f95874f1640 1 librados: setting wanted keys 2024-03-25T18:26:50.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.535+0000 7f95874f1640 1 librados: calling monclient init 2024-03-25T18:26:50.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.535+0000 7f95874f1640 10 monclient: init 2024-03-25T18:26:50.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.536+0000 7f95874f1640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:50.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.536+0000 7f95874f1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:26:50.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.536+0000 7f95874f1640 10 monclient(hunting): picked mon.noname-b con 0x7f95800b1b00 addr v1:172.21.15.16:6789/0 2024-03-25T18:26:50.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.536+0000 7f95874f1640 10 monclient(hunting): picked mon.noname-c con 0x7f95800b44b0 addr v1:172.21.15.3:6790/0 2024-03-25T18:26:50.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.536+0000 7f95874f1640 10 monclient(hunting): picked mon.noname-a con 0x7f95800b4e70 addr v1:172.21.15.3:6789/0 2024-03-25T18:26:50.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.536+0000 7f95874f1640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9580142140 con 0x7f95800b4e70 2024-03-25T18:26:50.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.536+0000 7f95874f1640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9580141d20 con 0x7f95800b1b00 2024-03-25T18:26:50.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.536+0000 7f95874f1640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9580140c10 con 0x7f95800b44b0 2024-03-25T18:26:50.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.536+0000 7f95874f1640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:50.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.536+0000 7f95874f1640 10 monclient(hunting): authenticate will time out at 7464.796500s 2024-03-25T18:26:50.550 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.549+0000 7f9584a8c640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f95800b44b0 0x7f9580134f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:39634/0 (socket says 172.21.15.3:39634) 2024-03-25T18:26:50.550 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.549+0000 7f9584a8c640 1 -- 172.21.15.3:0/1433740368 learned_addr learned my addr 172.21.15.3:0/1433740368 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:26:50.551 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.550+0000 7f9575ffb640 10 client.?.objecter ms_handle_connect 0x7f95800b44b0 2024-03-25T18:26:50.551 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.550+0000 7f9575ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:50.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.556+0000 7f9575ffb640 1 -- 172.21.15.3:0/1433740368 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 79602274 0 0) 0x7f9580140c10 con 0x7f95800b44b0 2024-03-25T18:26:50.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.556+0000 7f9575ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:50.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.556+0000 7f9575ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:50.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.556+0000 7f9575ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:50.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.556+0000 7f9575ffb640 10 monclient(hunting): my global_id is 4289 2024-03-25T18:26:50.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.556+0000 7f9575ffb640 1 -- 172.21.15.3:0/1433740368 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9564004eb0 con 0x7f95800b44b0 2024-03-25T18:26:50.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.557+0000 7f9575ffb640 10 client.?.objecter ms_handle_connect 0x7f95800b4e70 2024-03-25T18:26:50.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.557+0000 7f9575ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:50.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.557+0000 7f9575ffb640 10 client.?.objecter ms_handle_connect 0x7f95800b1b00 2024-03-25T18:26:50.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.557+0000 7f9575ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:50.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.563+0000 7f9575ffb640 1 -- 172.21.15.3:0/1433740368 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1290929862 0 0) 0x7f9580142140 con 0x7f95800b4e70 2024-03-25T18:26:50.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.563+0000 7f9575ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:50.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.563+0000 7f9575ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:50.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.563+0000 7f9575ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:50.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.563+0000 7f9575ffb640 10 monclient(hunting): my global_id is 4317 2024-03-25T18:26:50.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.563+0000 7f9575ffb640 1 -- 172.21.15.3:0/1433740368 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9580140c10 con 0x7f95800b4e70 2024-03-25T18:26:50.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.568+0000 7f9575ffb640 1 -- 172.21.15.3:0/1433740368 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3800467034 0 0) 0x7f9564004eb0 con 0x7f95800b44b0 2024-03-25T18:26:50.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.568+0000 7f9575ffb640 1 -- 172.21.15.3:0/1433740368 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9580142140 con 0x7f95800b44b0 2024-03-25T18:26:50.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.568+0000 7f9575ffb640 1 -- 172.21.15.3:0/1433740368 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f9570001950 con 0x7f95800b44b0 2024-03-25T18:26:50.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.568+0000 7f9575ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:50.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.568+0000 7f9575ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:50.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.568+0000 7f9575ffb640 10 monclient(hunting): dump: 2024-03-25T18:26:50.570 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:50.570 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:50.570 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:50.570 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:50.571 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:50.571 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:50.571 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:50.571 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:50.571 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:50.571 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:50.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.568+0000 7f9575ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:50.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.570+0000 7f9575ffb640 1 -- 172.21.15.3:0/1433740368 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018617715 0 0) 0x7f9580141d20 con 0x7f95800b1b00 2024-03-25T18:26:50.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.570+0000 7f9575ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:50.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.570+0000 7f9575ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:50.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.570+0000 7f9575ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:50.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.570+0000 7f9575ffb640 10 monclient(hunting): my global_id is 4279 2024-03-25T18:26:50.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.570+0000 7f9575ffb640 1 -- 172.21.15.3:0/1433740368 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9564004eb0 con 0x7f95800b1b00 2024-03-25T18:26:50.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.571+0000 7f9575ffb640 1 -- 172.21.15.3:0/1433740368 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 420126390 0 0) 0x7f9580140c10 con 0x7f95800b4e70 2024-03-25T18:26:50.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.571+0000 7f9575ffb640 1 -- 172.21.15.3:0/1433740368 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9580141d20 con 0x7f95800b4e70 2024-03-25T18:26:50.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.571+0000 7f9575ffb640 1 -- 172.21.15.3:0/1433740368 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f9578001cb0 con 0x7f95800b4e70 2024-03-25T18:26:50.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.571+0000 7f9575ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:50.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.571+0000 7f9575ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:50.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.571+0000 7f9575ffb640 10 monclient(hunting): dump: 2024-03-25T18:26:50.573 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:50.573 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:50.573 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:50.573 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:50.573 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:50.573 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:50.573 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:50.573 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:50.574 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:50.574 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:50.586 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.584+0000 7f9575ffb640 1 -- 172.21.15.3:0/1433740368 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 636642261 0 0) 0x7f9580141d20 con 0x7f95800b4e70 2024-03-25T18:26:50.586 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.584+0000 7f9575ffb640 1 -- 172.21.15.3:0/1433740368 >> v1:172.21.15.3:6790/0 conn(0x7f95800b44b0 legacy=0x7f9580134f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:50.586 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.584+0000 7f9575ffb640 1 -- 172.21.15.3:0/1433740368 >> v1:172.21.15.16:6789/0 conn(0x7f95800b1b00 legacy=0x7f95801403d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:50.586 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.584+0000 7f9575ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:26:50.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.584+0000 7f9575ffb640 1 monclient: found mon.a 2024-03-25T18:26:50.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.584+0000 7f9575ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:50.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.584+0000 7f9575ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:26:50.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.584+0000 7f9575ffb640 1 -- 172.21.15.3:0/1433740368 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f958013c470 con 0x7f95800b4e70 2024-03-25T18:26:50.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.584+0000 7f9575ffb640 10 monclient: _finish_auth 0 2024-03-25T18:26:50.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.584+0000 7f9575ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:26:50.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.584+0000 7f9575ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:50.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.584+0000 7f95874f1640 5 monclient: authenticate success, global_id 4317 2024-03-25T18:26:50.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.584+0000 7f95874f1640 10 monclient: _renew_subs 2024-03-25T18:26:50.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.584+0000 7f95874f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:26:50.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.584+0000 7f95874f1640 1 -- 172.21.15.3:0/1433740368 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f958013cc70 con 0x7f95800b4e70 2024-03-25T18:26:50.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.585+0000 7f95874f1640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:50.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.585+0000 7f95874f1640 10 monclient: _renew_subs 2024-03-25T18:26:50.588 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.585+0000 7f95874f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:26:50.588 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.585+0000 7f95874f1640 1 -- 172.21.15.3:0/1433740368 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9580135880 con 0x7f95800b4e70 2024-03-25T18:26:50.588 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.585+0000 7f95874f1640 1 librados: init done 2024-03-25T18:26:50.588 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.586+0000 7f95874f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:50.588 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.586+0000 7f95874f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:50.588 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.586+0000 7f95874f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:26:50.589 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.586+0000 7f95874f1640 1 -- 172.21.15.3:0/1433740368 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9580141b60 con 0x7f95800b4e70 2024-03-25T18:26:50.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.915+0000 7f9575ffb640 1 -- 172.21.15.3:0/1433740368 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95780020c0 con 0x7f95800b4e70 2024-03-25T18:26:50.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.915+0000 7f9575ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:50.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.915+0000 7f9575ffb640 1 -- 172.21.15.3:0/1433740368 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f9578003b30 con 0x7f95800b4e70 2024-03-25T18:26:50.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.915+0000 7f9575ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:50.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.915+0000 7f9575ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:50.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.915+0000 7f9575ffb640 10 monclient: dump: 2024-03-25T18:26:50.917 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:50.917 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:50.917 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:50.917 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:50.917 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:50.918 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:50.918 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:50.918 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:50.918 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:50.918 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:50.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.915+0000 7f9576ffd640 4 set_mon_vals no callback set 2024-03-25T18:26:50.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.916+0000 7f9575ffb640 1 -- 172.21.15.3:0/1433740368 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f9578003d50 con 0x7f95800b4e70 2024-03-25T18:26:50.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.917+0000 7f958528d640 10 monclient: get_auth_request con 0x7f956406df00 auth_method 0 2024-03-25T18:26:50.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.917+0000 7f9575ffb640 10 client.4317.objecter ms_handle_connect 0x7f956406df00 2024-03-25T18:26:50.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.927+0000 7f9575ffb640 1 -- 172.21.15.3:0/1433740368 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f9578053710 con 0x7f95800b4e70 2024-03-25T18:26:50.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.927+0000 7f9575ffb640 10 client.4317.objecter ms_dispatch 0x7f9580000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:50.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.927+0000 7f9575ffb640 3 client.4317.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:50.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.927+0000 7f9575ffb640 3 client.4317.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:50.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.927+0000 7f9575ffb640 20 client.4317.objecter dump_active .. 0 homeless 2024-03-25T18:26:50.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.927+0000 7f9575ffb640 1 -- 172.21.15.3:0/1433740368 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f957804f220 con 0x7f95800b4e70 2024-03-25T18:26:50.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.927+0000 7f9575ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:50.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:50.927+0000 7f9575ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:51.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:51.154+0000 7f95874f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:51.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:51.154+0000 7f95874f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:51.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:51.154+0000 7f95874f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:26:51.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:51.154+0000 7f95874f1640 1 -- 172.21.15.3:0/1433740368 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f958004f730 con 0x7f95800b4e70 2024-03-25T18:26:51.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:51.199+0000 7f9575ffb640 1 -- 172.21.15.3:0/1433740368 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f9578051150 con 0x7f95800b4e70 2024-03-25T18:26:51.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:51.199+0000 7f9575ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:51.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:51.199+0000 7f9575ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:51.201 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:26:51.201 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:51.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:51.201+0000 7f95874f1640 10 librados: watch_flush enter 2024-03-25T18:26:51.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:51.201+0000 7f95874f1640 10 librados: watch_flush exit 2024-03-25T18:26:51.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:51.201+0000 7f95874f1640 20 client.4317.objecter shutdown clearing up homeless session... 2024-03-25T18:26:51.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:51.201+0000 7f95874f1640 10 client.4317.objecter successfully canceled tick 2024-03-25T18:26:51.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:51.201+0000 7f95874f1640 1 -- 172.21.15.3:0/1433740368 >> v1:172.21.15.3:6800/4195140966 conn(0x7f956406df00 legacy=0x7f95640703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:51.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:51.201+0000 7f95874f1640 10 monclient: shutdown 2024-03-25T18:26:51.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:51.201+0000 7f95874f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:51.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:51.201+0000 7f95874f1640 1 -- 172.21.15.3:0/1433740368 >> v1:172.21.15.3:6789/0 conn(0x7f95800b4e70 legacy=0x7f95801dbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:51.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:51.201+0000 7f95874f1640 1 -- 172.21.15.3:0/1433740368 shutdown_connections 2024-03-25T18:26:51.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:51.201+0000 7f95874f1640 1 -- 172.21.15.3:0/1433740368 >> 172.21.15.3:0/1433740368 conn(0x7f95800617a0 msgr2=0x7f95800b6480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:51.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:51.201+0000 7f95874f1640 1 -- 172.21.15.3:0/1433740368 shutdown_connections 2024-03-25T18:26:51.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:51.201+0000 7f95874f1640 1 -- 172.21.15.3:0/1433740368 wait complete. 2024-03-25T18:26:51.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:51.201+0000 7f95874f1640 1 librados: shutdown 2024-03-25T18:26:51.215 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:58.023 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:58.143 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.142+0000 7fc364c63640 10 monclient: get_monmap_and_config 2024-03-25T18:26:58.143 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.142+0000 7fc364c63640 10 monclient: build_initial_monmap 2024-03-25T18:26:58.143 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.142+0000 7fc364c63640 10 monclient: monmap: 2024-03-25T18:26:58.143 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:26:58.143 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:58.143 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:26:58.143264+0000 2024-03-25T18:26:58.144 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:26:58.143264+0000 2024-03-25T18:26:58.144 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:58.144 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:58.144 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:26:58.144 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:26:58.144 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:26:58.144 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:58.145 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.142+0000 7fc364c63640 1 Processor -- start 2024-03-25T18:26:58.145 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.142+0000 7fc364c63640 1 -- start start 2024-03-25T18:26:58.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.142+0000 7fc364c63640 10 monclient: init 2024-03-25T18:26:58.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.143+0000 7fc364c63640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:58.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.143+0000 7fc364c63640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:58.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.143+0000 7fc364c63640 10 monclient(hunting): picked mon.noname-a con 0x7fc3600b4e70 addr v1:172.21.15.3:6789/0 2024-03-25T18:26:58.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.143+0000 7fc364c63640 10 monclient(hunting): picked mon.noname-c con 0x7fc3600b44b0 addr v1:172.21.15.16:6789/0 2024-03-25T18:26:58.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.143+0000 7fc364c63640 10 monclient(hunting): picked mon.noname-b con 0x7fc3600b1b00 addr v1:172.21.15.3:6790/0 2024-03-25T18:26:58.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.143+0000 7fc364c63640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3600a96f0 con 0x7fc3600b4e70 2024-03-25T18:26:58.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.143+0000 7fc364c63640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3600ab0e0 con 0x7fc3600b44b0 2024-03-25T18:26:58.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.143+0000 7fc364c63640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc360145f70 con 0x7fc3600b1b00 2024-03-25T18:26:58.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.143+0000 7fc364c63640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:58.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.143+0000 7fc364c63640 10 monclient(hunting): authenticate will time out at 7472.404286s 2024-03-25T18:26:58.149 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.147+0000 7fc35e59c640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fc3600b4e70 0x7fc36013f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:46112/0 (socket says 172.21.15.3:46112) 2024-03-25T18:26:58.149 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.147+0000 7fc35e59c640 1 -- 172.21.15.3:0/1707299801 learned_addr learned my addr 172.21.15.3:0/1707299801 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:26:58.160 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.159+0000 7fc35d59a640 1 -- 172.21.15.3:0/1707299801 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18054443 0 0) 0x7fc3600a96f0 con 0x7fc3600b4e70 2024-03-25T18:26:58.161 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.159+0000 7fc35d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:26:58.161 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.159+0000 7fc35d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:58.161 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.159+0000 7fc35d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:58.161 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.159+0000 7fc35d59a640 10 monclient(hunting): my global_id is 4320 2024-03-25T18:26:58.161 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.159+0000 7fc35d59a640 1 -- 172.21.15.3:0/1707299801 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc348003520 con 0x7fc3600b4e70 2024-03-25T18:26:58.172 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.171+0000 7fc35d59a640 1 -- 172.21.15.3:0/1707299801 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2886704174 0 0) 0x7fc360145f70 con 0x7fc3600b1b00 2024-03-25T18:26:58.173 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.171+0000 7fc35d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:26:58.173 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.171+0000 7fc35d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:58.173 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.171+0000 7fc35d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:58.173 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.171+0000 7fc35d59a640 10 monclient(hunting): my global_id is 4292 2024-03-25T18:26:58.173 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.171+0000 7fc35d59a640 1 -- 172.21.15.3:0/1707299801 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3600a96f0 con 0x7fc3600b1b00 2024-03-25T18:26:58.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.175+0000 7fc35d59a640 1 -- 172.21.15.3:0/1707299801 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484514881 0 0) 0x7fc3600ab0e0 con 0x7fc3600b44b0 2024-03-25T18:26:58.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.175+0000 7fc35d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:26:58.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.175+0000 7fc35d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:58.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.175+0000 7fc35d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:58.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.175+0000 7fc35d59a640 10 monclient(hunting): my global_id is 4282 2024-03-25T18:26:58.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.175+0000 7fc35d59a640 1 -- 172.21.15.3:0/1707299801 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc360145f70 con 0x7fc3600b44b0 2024-03-25T18:26:58.187 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.185+0000 7fc35d59a640 1 -- 172.21.15.3:0/1707299801 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2472823065 0 0) 0x7fc360145f70 con 0x7fc3600b44b0 2024-03-25T18:26:58.187 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.185+0000 7fc35d59a640 1 -- 172.21.15.3:0/1707299801 >> v1:172.21.15.3:6790/0 conn(0x7fc3600b1b00 legacy=0x7fc36014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:58.187 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.185+0000 7fc35d59a640 1 -- 172.21.15.3:0/1707299801 >> v1:172.21.15.3:6789/0 conn(0x7fc3600b4e70 legacy=0x7fc36013f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:58.187 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.185+0000 7fc35d59a640 10 monclient: _finish_hunting 0 2024-03-25T18:26:58.187 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.185+0000 7fc35d59a640 1 monclient: found mon.noname-c 2024-03-25T18:26:58.187 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.185+0000 7fc35d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:58.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.185+0000 7fc35d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:26:58.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.185+0000 7fc35d59a640 1 -- 172.21.15.3:0/1707299801 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3600a9b80 con 0x7fc3600b44b0 2024-03-25T18:26:58.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.185+0000 7fc35d59a640 10 monclient: _finish_auth 0 2024-03-25T18:26:58.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.185+0000 7fc35d59a640 10 monclient: _check_auth_tickets 2024-03-25T18:26:58.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.185+0000 7fc35d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:58.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.185+0000 7fc35d59a640 1 -- 172.21.15.3:0/1707299801 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc354001630 con 0x7fc3600b44b0 2024-03-25T18:26:58.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.185+0000 7fc35d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:58.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.185+0000 7fc35d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:58.189 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.185+0000 7fc35d59a640 10 monclient: dump: 2024-03-25T18:26:58.189 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:58.189 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:58.189 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:58.189 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:58.189 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:58.189 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:58.189 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:58.189 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:58.189 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:58.189 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:58.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.185+0000 7fc364c63640 5 monclient: authenticate success, global_id 4282 2024-03-25T18:26:58.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.185+0000 7fc364c63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:58.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.210+0000 7fc35d59a640 1 -- 172.21.15.3:0/1707299801 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc354001790 con 0x7fc3600b44b0 2024-03-25T18:26:58.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.210+0000 7fc35d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:58.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.210+0000 7fc35d59a640 1 -- 172.21.15.3:0/1707299801 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc354001be0 con 0x7fc3600b44b0 2024-03-25T18:26:58.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.210+0000 7fc35d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:58.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.210+0000 7fc35d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:58.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.210+0000 7fc35d59a640 10 monclient: dump: 2024-03-25T18:26:58.211 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:58.211 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:58.211 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:58.211 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:58.212 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:58.212 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:58.212 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:58.212 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:58.212 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:58.212 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:58.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.210+0000 7fc364c63640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:58.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.210+0000 7fc364c63640 4 set_mon_vals no callback set 2024-03-25T18:26:58.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.210+0000 7fc364c63640 10 monclient: shutdown 2024-03-25T18:26:58.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.210+0000 7fc364c63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:58.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.210+0000 7fc364c63640 1 -- 172.21.15.3:0/1707299801 >> v1:172.21.15.16:6789/0 conn(0x7fc3600b44b0 legacy=0x7fc360145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:58.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.210+0000 7fc364c63640 1 -- 172.21.15.3:0/1707299801 shutdown_connections 2024-03-25T18:26:58.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.211+0000 7fc364c63640 1 -- 172.21.15.3:0/1707299801 >> 172.21.15.3:0/1707299801 conn(0x7fc3600617a0 msgr2=0x7fc360061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:58.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.211+0000 7fc364c63640 1 -- 172.21.15.3:0/1707299801 shutdown_connections 2024-03-25T18:26:58.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.211+0000 7fc364c63640 1 -- 172.21.15.3:0/1707299801 wait complete. 2024-03-25T18:26:58.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.211+0000 7fc364c63640 10 monclient: build_initial_monmap 2024-03-25T18:26:58.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.211+0000 7fc364c63640 10 monclient: monmap: 2024-03-25T18:26:58.215 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:26:58.215 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:58.215 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:26:58.215 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:26:58.215 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:58.215 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:26:58.215 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:26:58.215 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:26:58.215 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:26:58.215 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:58.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.212+0000 7fc364c63640 1 librados: starting msgr at 2024-03-25T18:26:58.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.212+0000 7fc364c63640 1 librados: starting objecter 2024-03-25T18:26:58.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.212+0000 7fc364c63640 1 Processor -- start 2024-03-25T18:26:58.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.212+0000 7fc364c63640 1 -- start start 2024-03-25T18:26:58.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.212+0000 7fc364c63640 1 librados: setting wanted keys 2024-03-25T18:26:58.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.212+0000 7fc364c63640 1 librados: calling monclient init 2024-03-25T18:26:58.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.212+0000 7fc364c63640 10 monclient: init 2024-03-25T18:26:58.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.212+0000 7fc364c63640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:58.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.212+0000 7fc364c63640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:58.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.212+0000 7fc364c63640 10 monclient(hunting): picked mon.noname-a con 0x7fc3600b1b00 addr v1:172.21.15.3:6789/0 2024-03-25T18:26:58.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.212+0000 7fc364c63640 10 monclient(hunting): picked mon.noname-b con 0x7fc3600b44b0 addr v1:172.21.15.16:6789/0 2024-03-25T18:26:58.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.213+0000 7fc364c63640 10 monclient(hunting): picked mon.noname-c con 0x7fc3600b4e70 addr v1:172.21.15.3:6790/0 2024-03-25T18:26:58.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.213+0000 7fc364c63640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc360136b50 con 0x7fc3600b1b00 2024-03-25T18:26:58.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.213+0000 7fc364c63640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc360137eb0 con 0x7fc3600b44b0 2024-03-25T18:26:58.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.213+0000 7fc364c63640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc360134590 con 0x7fc3600b4e70 2024-03-25T18:26:58.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.213+0000 7fc364c63640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:58.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.213+0000 7fc364c63640 10 monclient(hunting): authenticate will time out at 7472.473406s 2024-03-25T18:26:58.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.225+0000 7fc35ed9d640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fc3600b4e70 0x7fc360133e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:43988/0 (socket says 172.21.15.3:43988) 2024-03-25T18:26:58.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.225+0000 7fc35ed9d640 1 -- 172.21.15.3:0/1468332505 learned_addr learned my addr 172.21.15.3:0/1468332505 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:26:58.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.226+0000 7fc34effd640 10 client.?.objecter ms_handle_connect 0x7fc3600b4e70 2024-03-25T18:26:58.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.226+0000 7fc34effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:58.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.228+0000 7fc34effd640 10 client.?.objecter ms_handle_connect 0x7fc3600b44b0 2024-03-25T18:26:58.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.228+0000 7fc34effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:58.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.230+0000 7fc34effd640 10 client.?.objecter ms_handle_connect 0x7fc3600b1b00 2024-03-25T18:26:58.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.230+0000 7fc34effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:58.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.239+0000 7fc34effd640 1 -- 172.21.15.3:0/1468332505 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 397019668 0 0) 0x7fc360134590 con 0x7fc3600b4e70 2024-03-25T18:26:58.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.239+0000 7fc34effd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:58.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.239+0000 7fc34effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:58.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.239+0000 7fc34effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:58.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.239+0000 7fc34effd640 10 monclient(hunting): my global_id is 4295 2024-03-25T18:26:58.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.239+0000 7fc34effd640 1 -- 172.21.15.3:0/1468332505 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc340004eb0 con 0x7fc3600b4e70 2024-03-25T18:26:58.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.239+0000 7fc34effd640 1 -- 172.21.15.3:0/1468332505 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170950751 0 0) 0x7fc360136b50 con 0x7fc3600b1b00 2024-03-25T18:26:58.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.239+0000 7fc34effd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:58.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.239+0000 7fc34effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:58.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.239+0000 7fc34effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:58.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.239+0000 7fc34effd640 10 monclient(hunting): my global_id is 4323 2024-03-25T18:26:58.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.240+0000 7fc34effd640 1 -- 172.21.15.3:0/1468332505 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc360134590 con 0x7fc3600b1b00 2024-03-25T18:26:58.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.242+0000 7fc34effd640 1 -- 172.21.15.3:0/1468332505 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230544509 0 0) 0x7fc360137eb0 con 0x7fc3600b44b0 2024-03-25T18:26:58.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.242+0000 7fc34effd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:58.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.242+0000 7fc34effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:58.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.242+0000 7fc34effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:58.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.242+0000 7fc34effd640 10 monclient(hunting): my global_id is 4285 2024-03-25T18:26:58.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.242+0000 7fc34effd640 1 -- 172.21.15.3:0/1468332505 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc360136b50 con 0x7fc3600b44b0 2024-03-25T18:26:58.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.263+0000 7fc34effd640 1 -- 172.21.15.3:0/1468332505 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1235212285 0 0) 0x7fc360134590 con 0x7fc3600b1b00 2024-03-25T18:26:58.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.263+0000 7fc34effd640 1 -- 172.21.15.3:0/1468332505 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc360137eb0 con 0x7fc3600b1b00 2024-03-25T18:26:58.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.263+0000 7fc34effd640 1 -- 172.21.15.3:0/1468332505 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc3500017b0 con 0x7fc3600b1b00 2024-03-25T18:26:58.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.263+0000 7fc34effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:58.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.263+0000 7fc34effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:58.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.263+0000 7fc34effd640 10 monclient(hunting): dump: 2024-03-25T18:26:58.265 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:58.265 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:58.265 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:58.265 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:58.265 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:58.265 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:58.265 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:58.265 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:58.265 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:58.265 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:58.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.263+0000 7fc34effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:58.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.266+0000 7fc34effd640 1 -- 172.21.15.3:0/1468332505 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2835083375 0 0) 0x7fc340004eb0 con 0x7fc3600b4e70 2024-03-25T18:26:58.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.266+0000 7fc34effd640 1 -- 172.21.15.3:0/1468332505 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc360134590 con 0x7fc3600b4e70 2024-03-25T18:26:58.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.266+0000 7fc34effd640 1 -- 172.21.15.3:0/1468332505 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc358001d10 con 0x7fc3600b4e70 2024-03-25T18:26:58.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.266+0000 7fc34effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:58.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.266+0000 7fc34effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:58.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.266+0000 7fc34effd640 10 monclient(hunting): dump: 2024-03-25T18:26:58.267 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:58.267 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:58.267 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:58.267 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:58.267 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:58.267 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:58.267 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:58.268 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:58.268 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:58.268 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:58.275 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.274+0000 7fc34effd640 1 -- 172.21.15.3:0/1468332505 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3489837403 0 0) 0x7fc360137eb0 con 0x7fc3600b1b00 2024-03-25T18:26:58.275 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.274+0000 7fc34effd640 1 -- 172.21.15.3:0/1468332505 >> v1:172.21.15.3:6790/0 conn(0x7fc3600b4e70 legacy=0x7fc360133e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:58.275 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.274+0000 7fc34effd640 1 -- 172.21.15.3:0/1468332505 >> v1:172.21.15.16:6789/0 conn(0x7fc3600b44b0 legacy=0x7fc360136260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:58.275 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.274+0000 7fc34effd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:58.275 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.274+0000 7fc34effd640 1 monclient: found mon.a 2024-03-25T18:26:58.275 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.274+0000 7fc34effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:58.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.274+0000 7fc34effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:26:58.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.274+0000 7fc34effd640 1 -- 172.21.15.3:0/1468332505 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc360134720 con 0x7fc3600b1b00 2024-03-25T18:26:58.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.274+0000 7fc34effd640 10 monclient: _finish_auth 0 2024-03-25T18:26:58.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.274+0000 7fc34effd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:58.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.274+0000 7fc34effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:58.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.274+0000 7fc364c63640 5 monclient: authenticate success, global_id 4323 2024-03-25T18:26:58.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.274+0000 7fc364c63640 10 monclient: _renew_subs 2024-03-25T18:26:58.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.274+0000 7fc364c63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:26:58.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.274+0000 7fc364c63640 1 -- 172.21.15.3:0/1468332505 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc360134970 con 0x7fc3600b1b00 2024-03-25T18:26:58.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.274+0000 7fc364c63640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:58.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.274+0000 7fc364c63640 10 monclient: _renew_subs 2024-03-25T18:26:58.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.274+0000 7fc364c63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:26:58.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.274+0000 7fc364c63640 1 -- 172.21.15.3:0/1468332505 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3601daf30 con 0x7fc3600b1b00 2024-03-25T18:26:58.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.274+0000 7fc364c63640 1 librados: init done 2024-03-25T18:26:58.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.275+0000 7fc364c63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:58.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.275+0000 7fc364c63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:58.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.275+0000 7fc364c63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:26:58.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.275+0000 7fc364c63640 1 -- 172.21.15.3:0/1468332505 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc36004f730 con 0x7fc3600b1b00 2024-03-25T18:26:58.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.613+0000 7fc34effd640 1 -- 172.21.15.3:0/1468332505 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc350001d20 con 0x7fc3600b1b00 2024-03-25T18:26:58.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.613+0000 7fc34effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:58.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.613+0000 7fc34effd640 1 -- 172.21.15.3:0/1468332505 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc350004980 con 0x7fc3600b1b00 2024-03-25T18:26:58.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.613+0000 7fc34effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:58.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.613+0000 7fc34effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:58.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.613+0000 7fc34effd640 10 monclient: dump: 2024-03-25T18:26:58.615 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:26:58.615 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:26:58.615 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:58.615 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:26:58.615 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:58.615 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:26:58.615 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:26:58.615 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:26:58.615 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:26:58.615 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:26:58.616 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.613+0000 7fc35cd99640 4 set_mon_vals no callback set 2024-03-25T18:26:58.616 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.614+0000 7fc34effd640 1 -- 172.21.15.3:0/1468332505 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7fc35001a950 con 0x7fc3600b1b00 2024-03-25T18:26:58.616 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.615+0000 7fc35dd9b640 10 monclient: get_auth_request con 0x7fc34006dec0 auth_method 0 2024-03-25T18:26:58.616 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.615+0000 7fc34effd640 10 client.4323.objecter ms_handle_connect 0x7fc34006dec0 2024-03-25T18:26:58.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.621+0000 7fc34effd640 1 -- 172.21.15.3:0/1468332505 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7fc3500546c0 con 0x7fc3600b1b00 2024-03-25T18:26:58.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.621+0000 7fc34effd640 10 client.4323.objecter ms_dispatch 0x7fc360000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:58.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.621+0000 7fc34effd640 3 client.4323.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:58.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.621+0000 7fc34effd640 3 client.4323.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:58.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.621+0000 7fc34effd640 20 client.4323.objecter dump_active .. 0 homeless 2024-03-25T18:26:58.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.621+0000 7fc34effd640 1 -- 172.21.15.3:0/1468332505 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc350050000 con 0x7fc3600b1b00 2024-03-25T18:26:58.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.621+0000 7fc34effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:58.623 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.621+0000 7fc34effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:58.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.845+0000 7fc364c63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:58.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.845+0000 7fc364c63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:58.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.845+0000 7fc364c63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:26:58.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.845+0000 7fc364c63640 1 -- 172.21.15.3:0/1468332505 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3600aa090 con 0x7fc3600b1b00 2024-03-25T18:26:58.904 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.902+0000 7fc34effd640 1 -- 172.21.15.3:0/1468332505 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7fc350052100 con 0x7fc3600b1b00 2024-03-25T18:26:58.904 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.903+0000 7fc34effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:58.904 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.903+0000 7fc34effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:58.904 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:26:58.904 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:58.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.905+0000 7fc364c63640 10 librados: watch_flush enter 2024-03-25T18:26:58.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.905+0000 7fc364c63640 10 librados: watch_flush exit 2024-03-25T18:26:58.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.905+0000 7fc364c63640 20 client.4323.objecter shutdown clearing up homeless session... 2024-03-25T18:26:58.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.905+0000 7fc364c63640 10 client.4323.objecter successfully canceled tick 2024-03-25T18:26:58.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.905+0000 7fc364c63640 1 -- 172.21.15.3:0/1468332505 >> v1:172.21.15.3:6800/4195140966 conn(0x7fc34006dec0 legacy=0x7fc340070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:58.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.905+0000 7fc364c63640 10 monclient: shutdown 2024-03-25T18:26:58.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.905+0000 7fc364c63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:58.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.905+0000 7fc364c63640 1 -- 172.21.15.3:0/1468332505 >> v1:172.21.15.3:6789/0 conn(0x7fc3600b1b00 legacy=0x7fc360135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:58.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.905+0000 7fc364c63640 1 -- 172.21.15.3:0/1468332505 shutdown_connections 2024-03-25T18:26:58.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.905+0000 7fc364c63640 1 -- 172.21.15.3:0/1468332505 >> 172.21.15.3:0/1468332505 conn(0x7fc3600617a0 msgr2=0x7fc360144a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:58.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.905+0000 7fc364c63640 1 -- 172.21.15.3:0/1468332505 shutdown_connections 2024-03-25T18:26:58.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.905+0000 7fc364c63640 1 -- 172.21.15.3:0/1468332505 wait complete. 2024-03-25T18:26:58.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:26:58.905+0000 7fc364c63640 1 librados: shutdown 2024-03-25T18:26:58.918 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:01.015 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:02:14.743 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:01.064 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:02:14.793 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:01.097 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:02:14.826 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:01.130 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:02:14.822 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:01.146 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:02:14.838 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:01.181 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:02:14.872 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:01.223 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:02:14.952 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:01.239 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:02:14.968 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:01.272 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:02:15.002 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:05.726 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:05.840 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.838+0000 7ff523f3d640 10 monclient: get_monmap_and_config 2024-03-25T18:27:05.840 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.838+0000 7ff523f3d640 10 monclient: build_initial_monmap 2024-03-25T18:27:05.840 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.839+0000 7ff523f3d640 10 monclient: monmap: 2024-03-25T18:27:05.840 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:27:05.840 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:05.840 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:27:05.840096+0000 2024-03-25T18:27:05.840 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:27:05.840096+0000 2024-03-25T18:27:05.840 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:05.840 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:05.840 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:27:05.841 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:27:05.841 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:27:05.841 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:05.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.839+0000 7ff523f3d640 1 Processor -- start 2024-03-25T18:27:05.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.839+0000 7ff523f3d640 1 -- start start 2024-03-25T18:27:05.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.839+0000 7ff523f3d640 10 monclient: init 2024-03-25T18:27:05.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.840+0000 7ff523f3d640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:05.843 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.840+0000 7ff523f3d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:27:05.843 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.840+0000 7ff523f3d640 10 monclient(hunting): picked mon.noname-c con 0x7ff51c0b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:27:05.843 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.840+0000 7ff523f3d640 10 monclient(hunting): picked mon.noname-a con 0x7ff51c0b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:27:05.843 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.840+0000 7ff523f3d640 10 monclient(hunting): picked mon.noname-b con 0x7ff51c13c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:27:05.843 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.840+0000 7ff523f3d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff51c0ab0e0 con 0x7ff51c0b4670 2024-03-25T18:27:05.843 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.840+0000 7ff523f3d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff51c0ab3c0 con 0x7ff51c0b4ff0 2024-03-25T18:27:05.843 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.840+0000 7ff523f3d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff51c0a9f30 con 0x7ff51c13c480 2024-03-25T18:27:05.843 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.840+0000 7ff523f3d640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:05.843 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.840+0000 7ff523f3d640 10 monclient(hunting): authenticate will time out at 7480.100754s 2024-03-25T18:27:05.843 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.840+0000 7ff5224da640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7ff51c13c480 0x7ff51c14dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:58094/0 (socket says 172.21.15.3:58094) 2024-03-25T18:27:05.843 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.840+0000 7ff5224da640 1 -- 172.21.15.3:0/1923995083 learned_addr learned my addr 172.21.15.3:0/1923995083 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:27:05.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.870+0000 7ff520cd7640 1 -- 172.21.15.3:0/1923995083 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2608128861 0 0) 0x7ff51c0ab0e0 con 0x7ff51c0b4670 2024-03-25T18:27:05.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.870+0000 7ff520cd7640 10 monclient(hunting): _negotiate 2024-03-25T18:27:05.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.870+0000 7ff520cd7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:05.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.870+0000 7ff520cd7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:05.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.870+0000 7ff520cd7640 10 monclient(hunting): my global_id is 4326 2024-03-25T18:27:05.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.870+0000 7ff520cd7640 1 -- 172.21.15.3:0/1923995083 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff508003520 con 0x7ff51c0b4670 2024-03-25T18:27:05.881 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.879+0000 7ff520cd7640 1 -- 172.21.15.3:0/1923995083 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2827290899 0 0) 0x7ff508003520 con 0x7ff51c0b4670 2024-03-25T18:27:05.881 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.879+0000 7ff520cd7640 1 -- 172.21.15.3:0/1923995083 >> v1:172.21.15.3:6790/0 conn(0x7ff51c13c480 legacy=0x7ff51c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:05.881 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.879+0000 7ff520cd7640 1 -- 172.21.15.3:0/1923995083 >> v1:172.21.15.16:6789/0 conn(0x7ff51c0b4ff0 legacy=0x7ff51c13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:27:05.881 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.879+0000 7ff520cd7640 10 monclient: _finish_hunting 0 2024-03-25T18:27:05.881 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.879+0000 7ff520cd7640 1 monclient: found mon.noname-a 2024-03-25T18:27:05.881 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.879+0000 7ff520cd7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:05.881 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.879+0000 7ff520cd7640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:27:05.881 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.879+0000 7ff520cd7640 1 -- 172.21.15.3:0/1923995083 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff51c0a9b70 con 0x7ff51c0b4670 2024-03-25T18:27:05.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.879+0000 7ff520cd7640 10 monclient: _finish_auth 0 2024-03-25T18:27:05.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.879+0000 7ff520cd7640 10 monclient: _check_auth_tickets 2024-03-25T18:27:05.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.879+0000 7ff520cd7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:05.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.879+0000 7ff523f3d640 5 monclient: authenticate success, global_id 4326 2024-03-25T18:27:05.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.879+0000 7ff520cd7640 1 -- 172.21.15.3:0/1923995083 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7ff50c001690 con 0x7ff51c0b4670 2024-03-25T18:27:05.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.879+0000 7ff520cd7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:05.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.879+0000 7ff520cd7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:05.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.879+0000 7ff520cd7640 10 monclient: dump: 2024-03-25T18:27:05.882 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:05.882 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:05.882 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:05.882 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:05.882 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:05.882 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:05.883 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:05.883 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:05.883 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:05.883 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:05.883 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.880+0000 7ff523f3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:05.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.884+0000 7ff520cd7640 1 -- 172.21.15.3:0/1923995083 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff50c001860 con 0x7ff51c0b4670 2024-03-25T18:27:05.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.884+0000 7ff520cd7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:05.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.884+0000 7ff520cd7640 1 -- 172.21.15.3:0/1923995083 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7ff50c001cb0 con 0x7ff51c0b4670 2024-03-25T18:27:05.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.884+0000 7ff520cd7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:05.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.884+0000 7ff520cd7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:05.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.884+0000 7ff520cd7640 10 monclient: dump: 2024-03-25T18:27:05.885 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:05.885 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:05.885 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:05.886 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:05.886 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:05.886 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:05.886 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:05.886 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:05.886 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:05.886 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:05.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.884+0000 7ff523f3d640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:05.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.884+0000 7ff523f3d640 4 set_mon_vals no callback set 2024-03-25T18:27:05.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.885+0000 7ff523f3d640 10 monclient: shutdown 2024-03-25T18:27:05.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.885+0000 7ff523f3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:05.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.885+0000 7ff523f3d640 1 -- 172.21.15.3:0/1923995083 >> v1:172.21.15.3:6789/0 conn(0x7ff51c0b4670 legacy=0x7ff51c1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:05.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.885+0000 7ff523f3d640 1 -- 172.21.15.3:0/1923995083 shutdown_connections 2024-03-25T18:27:05.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.885+0000 7ff523f3d640 1 -- 172.21.15.3:0/1923995083 >> 172.21.15.3:0/1923995083 conn(0x7ff51c0617a0 msgr2=0x7ff51c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:05.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.885+0000 7ff523f3d640 1 -- 172.21.15.3:0/1923995083 shutdown_connections 2024-03-25T18:27:05.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.885+0000 7ff523f3d640 1 -- 172.21.15.3:0/1923995083 wait complete. 2024-03-25T18:27:05.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.886+0000 7ff523f3d640 10 monclient: build_initial_monmap 2024-03-25T18:27:05.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.886+0000 7ff523f3d640 10 monclient: monmap: 2024-03-25T18:27:05.889 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:27:05.890 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:05.890 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:27:05.890 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:27:05.890 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:05.890 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:27:05.890 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:27:05.890 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:27:05.890 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:27:05.890 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:05.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.886+0000 7ff523f3d640 1 librados: starting msgr at 2024-03-25T18:27:05.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.886+0000 7ff523f3d640 1 librados: starting objecter 2024-03-25T18:27:05.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.886+0000 7ff523f3d640 1 Processor -- start 2024-03-25T18:27:05.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.886+0000 7ff523f3d640 1 -- start start 2024-03-25T18:27:05.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.886+0000 7ff523f3d640 1 librados: setting wanted keys 2024-03-25T18:27:05.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.886+0000 7ff523f3d640 1 librados: calling monclient init 2024-03-25T18:27:05.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.886+0000 7ff523f3d640 10 monclient: init 2024-03-25T18:27:05.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.887+0000 7ff523f3d640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:05.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.887+0000 7ff523f3d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:27:05.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.887+0000 7ff523f3d640 10 monclient(hunting): picked mon.noname-a con 0x7ff51c0b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:27:05.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.887+0000 7ff523f3d640 10 monclient(hunting): picked mon.noname-c con 0x7ff51c0b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:27:05.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.887+0000 7ff523f3d640 10 monclient(hunting): picked mon.noname-b con 0x7ff51c13c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:27:05.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.887+0000 7ff523f3d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff51c130cc0 con 0x7ff51c0b4670 2024-03-25T18:27:05.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.887+0000 7ff523f3d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff51c13d840 con 0x7ff51c13c480 2024-03-25T18:27:05.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.887+0000 7ff523f3d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff51c13db20 con 0x7ff51c0b4ff0 2024-03-25T18:27:05.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.887+0000 7ff523f3d640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:05.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.887+0000 7ff523f3d640 10 monclient(hunting): authenticate will time out at 7480.147682s 2024-03-25T18:27:05.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.896+0000 7ff521cd9640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ff51c0b4670 0x7ff51c13e590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:42590/0 (socket says 172.21.15.3:42590) 2024-03-25T18:27:05.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.896+0000 7ff521cd9640 1 -- 172.21.15.3:0/698058320 learned_addr learned my addr 172.21.15.3:0/698058320 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:27:05.900 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.899+0000 7ff5127fc640 10 client.?.objecter ms_handle_connect 0x7ff51c0b4670 2024-03-25T18:27:05.900 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.899+0000 7ff5127fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:05.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.906+0000 7ff5127fc640 10 client.?.objecter ms_handle_connect 0x7ff51c13c480 2024-03-25T18:27:05.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.906+0000 7ff5127fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:05.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.913+0000 7ff5127fc640 10 client.?.objecter ms_handle_connect 0x7ff51c0b4ff0 2024-03-25T18:27:05.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.913+0000 7ff5127fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:05.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.921+0000 7ff5127fc640 1 -- 172.21.15.3:0/698058320 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3610739542 0 0) 0x7ff51c130cc0 con 0x7ff51c0b4670 2024-03-25T18:27:05.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.921+0000 7ff5127fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:05.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.921+0000 7ff5127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:05.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.921+0000 7ff5127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:05.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.921+0000 7ff5127fc640 10 monclient(hunting): my global_id is 4329 2024-03-25T18:27:05.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:05.921+0000 7ff5127fc640 1 -- 172.21.15.3:0/698058320 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff500004eb0 con 0x7ff51c0b4670 2024-03-25T18:27:06.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.004+0000 7ff5127fc640 1 -- 172.21.15.3:0/698058320 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1685209810 0 0) 0x7ff51c13d840 con 0x7ff51c13c480 2024-03-25T18:27:06.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.004+0000 7ff5127fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:06.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.004+0000 7ff5127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:06.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.004+0000 7ff5127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:06.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.004+0000 7ff5127fc640 10 monclient(hunting): my global_id is 4288 2024-03-25T18:27:06.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.004+0000 7ff5127fc640 1 -- 172.21.15.3:0/698058320 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff51c130cc0 con 0x7ff51c13c480 2024-03-25T18:27:06.015 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.014+0000 7ff5127fc640 1 -- 172.21.15.3:0/698058320 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2978832373 0 0) 0x7ff500004eb0 con 0x7ff51c0b4670 2024-03-25T18:27:06.015 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.014+0000 7ff5127fc640 1 -- 172.21.15.3:0/698058320 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff51c13d840 con 0x7ff51c0b4670 2024-03-25T18:27:06.015 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.014+0000 7ff5127fc640 1 -- 172.21.15.3:0/698058320 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7ff518001960 con 0x7ff51c0b4670 2024-03-25T18:27:06.015 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.014+0000 7ff5127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:06.015 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.014+0000 7ff5127fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:06.015 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.014+0000 7ff5127fc640 10 monclient(hunting): dump: 2024-03-25T18:27:06.015 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:06.015 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:06.015 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:06.015 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:06.015 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:06.016 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:06.016 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:06.016 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:06.016 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:06.016 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:06.016 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.014+0000 7ff5127fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:06.016 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.015+0000 7ff5127fc640 1 -- 172.21.15.3:0/698058320 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 193340499 0 0) 0x7ff51c13db20 con 0x7ff51c0b4ff0 2024-03-25T18:27:06.016 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.015+0000 7ff5127fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:06.017 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.015+0000 7ff5127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:06.017 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.015+0000 7ff5127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:06.017 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.015+0000 7ff5127fc640 10 monclient(hunting): my global_id is 4301 2024-03-25T18:27:06.017 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.015+0000 7ff5127fc640 1 -- 172.21.15.3:0/698058320 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff500004eb0 con 0x7ff51c0b4ff0 2024-03-25T18:27:06.028 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.027+0000 7ff5127fc640 1 -- 172.21.15.3:0/698058320 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1954943490 0 0) 0x7ff500004eb0 con 0x7ff51c0b4ff0 2024-03-25T18:27:06.028 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.027+0000 7ff5127fc640 1 -- 172.21.15.3:0/698058320 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff51c13db20 con 0x7ff51c0b4ff0 2024-03-25T18:27:06.028 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.027+0000 7ff5127fc640 1 -- 172.21.15.3:0/698058320 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7ff50c001b70 con 0x7ff51c0b4ff0 2024-03-25T18:27:06.028 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.027+0000 7ff5127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:06.029 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.027+0000 7ff5127fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:06.029 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.027+0000 7ff5127fc640 10 monclient(hunting): dump: 2024-03-25T18:27:06.029 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:06.029 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:06.029 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:06.029 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:06.029 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:06.029 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:06.029 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:06.029 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:06.029 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:06.029 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:06.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.031+0000 7ff5127fc640 1 -- 172.21.15.3:0/698058320 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1236483293 0 0) 0x7ff51c130cc0 con 0x7ff51c13c480 2024-03-25T18:27:06.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.031+0000 7ff5127fc640 1 -- 172.21.15.3:0/698058320 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff500004eb0 con 0x7ff51c13c480 2024-03-25T18:27:06.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.031+0000 7ff5127fc640 1 -- 172.21.15.3:0/698058320 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7ff514001b00 con 0x7ff51c13c480 2024-03-25T18:27:06.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.031+0000 7ff5127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:06.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.031+0000 7ff5127fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:06.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.031+0000 7ff5127fc640 10 monclient(hunting): dump: 2024-03-25T18:27:06.033 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:06.033 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:06.033 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:06.033 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:06.033 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:06.033 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:06.033 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:06.033 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:06.033 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:06.033 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:06.035 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.032+0000 7ff5127fc640 1 -- 172.21.15.3:0/698058320 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 66402711 0 0) 0x7ff51c13d840 con 0x7ff51c0b4670 2024-03-25T18:27:06.035 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.032+0000 7ff5127fc640 1 -- 172.21.15.3:0/698058320 >> v1:172.21.15.3:6790/0 conn(0x7ff51c0b4ff0 legacy=0x7ff51c143660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:06.035 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.032+0000 7ff5127fc640 1 -- 172.21.15.3:0/698058320 >> v1:172.21.15.16:6789/0 conn(0x7ff51c13c480 legacy=0x7ff51c143d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:06.035 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.032+0000 7ff5127fc640 10 monclient: _finish_hunting 0 2024-03-25T18:27:06.035 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.032+0000 7ff5127fc640 1 monclient: found mon.a 2024-03-25T18:27:06.035 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.032+0000 7ff5127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:06.035 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.032+0000 7ff5127fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:27:06.035 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.032+0000 7ff5127fc640 1 -- 172.21.15.3:0/698058320 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff51c13cd10 con 0x7ff51c0b4670 2024-03-25T18:27:06.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.032+0000 7ff5127fc640 10 monclient: _finish_auth 0 2024-03-25T18:27:06.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.032+0000 7ff5127fc640 10 monclient: _check_auth_tickets 2024-03-25T18:27:06.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.032+0000 7ff5127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:06.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.032+0000 7ff523f3d640 5 monclient: authenticate success, global_id 4329 2024-03-25T18:27:06.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.032+0000 7ff523f3d640 10 monclient: _renew_subs 2024-03-25T18:27:06.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.032+0000 7ff523f3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:27:06.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.032+0000 7ff523f3d640 1 -- 172.21.15.3:0/698058320 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff51c13c930 con 0x7ff51c0b4670 2024-03-25T18:27:06.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.032+0000 7ff523f3d640 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:06.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.032+0000 7ff523f3d640 10 monclient: _renew_subs 2024-03-25T18:27:06.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.032+0000 7ff523f3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:27:06.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.032+0000 7ff523f3d640 1 -- 172.21.15.3:0/698058320 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff51c146360 con 0x7ff51c0b4670 2024-03-25T18:27:06.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.032+0000 7ff523f3d640 1 librados: init done 2024-03-25T18:27:06.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.033+0000 7ff523f3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:06.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.033+0000 7ff523f3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:06.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.033+0000 7ff523f3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:27:06.037 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.033+0000 7ff523f3d640 1 -- 172.21.15.3:0/698058320 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff51c13cb70 con 0x7ff51c0b4670 2024-03-25T18:27:06.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.394+0000 7ff5127fc640 1 -- 172.21.15.3:0/698058320 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5180035f0 con 0x7ff51c0b4670 2024-03-25T18:27:06.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.394+0000 7ff5127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:06.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.394+0000 7ff5127fc640 1 -- 172.21.15.3:0/698058320 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7ff5180037a0 con 0x7ff51c0b4670 2024-03-25T18:27:06.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.394+0000 7ff5127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:06.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.394+0000 7ff5127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:06.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.394+0000 7ff5127fc640 10 monclient: dump: 2024-03-25T18:27:06.396 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:06.396 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:06.396 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:06.396 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:06.396 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:06.396 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:06.397 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:06.397 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:06.397 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:06.397 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:06.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.394+0000 7ff513fff640 4 set_mon_vals no callback set 2024-03-25T18:27:06.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.395+0000 7ff5127fc640 1 -- 172.21.15.3:0/698058320 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7ff5180039c0 con 0x7ff51c0b4670 2024-03-25T18:27:06.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.396+0000 7ff5214d8640 10 monclient: get_auth_request con 0x7ff50006e110 auth_method 0 2024-03-25T18:27:06.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.396+0000 7ff5127fc640 10 client.4329.objecter ms_handle_connect 0x7ff50006e110 2024-03-25T18:27:06.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.402+0000 7ff5127fc640 1 -- 172.21.15.3:0/698058320 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7ff5180545f0 con 0x7ff51c0b4670 2024-03-25T18:27:06.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.402+0000 7ff5127fc640 10 client.4329.objecter ms_dispatch 0x7ff51c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:06.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.402+0000 7ff5127fc640 3 client.4329.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:06.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.402+0000 7ff5127fc640 3 client.4329.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:06.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.402+0000 7ff5127fc640 20 client.4329.objecter dump_active .. 0 homeless 2024-03-25T18:27:06.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.402+0000 7ff5127fc640 1 -- 172.21.15.3:0/698058320 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff518050100 con 0x7ff51c0b4670 2024-03-25T18:27:06.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.402+0000 7ff5127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:06.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.402+0000 7ff5127fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:06.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.659+0000 7ff523f3d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:06.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.659+0000 7ff523f3d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:06.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.659+0000 7ff523f3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:27:06.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.659+0000 7ff523f3d640 1 -- 172.21.15.3:0/698058320 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff51c06dbd0 con 0x7ff51c0b4670 2024-03-25T18:27:06.716 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.714+0000 7ff5127fc640 1 -- 172.21.15.3:0/698058320 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7ff518052030 con 0x7ff51c0b4670 2024-03-25T18:27:06.716 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.714+0000 7ff5127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:06.716 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.714+0000 7ff5127fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:06.716 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:27:06.716 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:06.718 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.717+0000 7ff523f3d640 10 librados: watch_flush enter 2024-03-25T18:27:06.718 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.717+0000 7ff523f3d640 10 librados: watch_flush exit 2024-03-25T18:27:06.718 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.717+0000 7ff523f3d640 20 client.4329.objecter shutdown clearing up homeless session... 2024-03-25T18:27:06.719 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.717+0000 7ff523f3d640 10 client.4329.objecter successfully canceled tick 2024-03-25T18:27:06.719 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.717+0000 7ff523f3d640 1 -- 172.21.15.3:0/698058320 >> v1:172.21.15.3:6800/4195140966 conn(0x7ff50006e110 legacy=0x7ff5000705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:06.719 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.717+0000 7ff523f3d640 10 monclient: shutdown 2024-03-25T18:27:06.719 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.717+0000 7ff523f3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:06.719 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.717+0000 7ff523f3d640 1 -- 172.21.15.3:0/698058320 >> v1:172.21.15.3:6789/0 conn(0x7ff51c0b4670 legacy=0x7ff51c13e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:06.719 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.717+0000 7ff523f3d640 1 -- 172.21.15.3:0/698058320 shutdown_connections 2024-03-25T18:27:06.719 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.717+0000 7ff523f3d640 1 -- 172.21.15.3:0/698058320 >> 172.21.15.3:0/698058320 conn(0x7ff51c0617a0 msgr2=0x7ff51c0b7030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:06.719 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.717+0000 7ff523f3d640 1 -- 172.21.15.3:0/698058320 shutdown_connections 2024-03-25T18:27:06.719 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.717+0000 7ff523f3d640 1 -- 172.21.15.3:0/698058320 wait complete. 2024-03-25T18:27:06.719 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:06.717+0000 7ff523f3d640 1 librados: shutdown 2024-03-25T18:27:06.731 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:13.538 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:13.657 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.655+0000 7fafafc3e640 10 monclient: get_monmap_and_config 2024-03-25T18:27:13.657 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.655+0000 7fafafc3e640 10 monclient: build_initial_monmap 2024-03-25T18:27:13.657 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.655+0000 7fafafc3e640 10 monclient: monmap: 2024-03-25T18:27:13.657 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:27:13.657 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:13.657 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:27:13.657046+0000 2024-03-25T18:27:13.657 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:27:13.657046+0000 2024-03-25T18:27:13.657 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:13.657 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:13.658 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:27:13.658 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:27:13.658 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:27:13.658 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:13.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.656+0000 7fafafc3e640 1 Processor -- start 2024-03-25T18:27:13.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.656+0000 7fafafc3e640 1 -- start start 2024-03-25T18:27:13.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.656+0000 7fafafc3e640 10 monclient: init 2024-03-25T18:27:13.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.657+0000 7fafafc3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:13.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.657+0000 7fafafc3e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:27:13.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.657+0000 7fafafc3e640 10 monclient(hunting): picked mon.noname-b con 0x7fafa80b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:27:13.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.657+0000 7fafafc3e640 10 monclient(hunting): picked mon.noname-a con 0x7fafa80b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:27:13.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.657+0000 7fafafc3e640 10 monclient(hunting): picked mon.noname-c con 0x7fafa813c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:27:13.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.657+0000 7fafafc3e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa80ab0e0 con 0x7fafa80b4670 2024-03-25T18:27:13.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.657+0000 7fafafc3e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa80ab3c0 con 0x7fafa813c480 2024-03-25T18:27:13.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.657+0000 7fafafc3e640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa80a9f30 con 0x7fafa80b4ff0 2024-03-25T18:27:13.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.657+0000 7fafafc3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:13.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.657+0000 7fafafc3e640 10 monclient(hunting): authenticate will time out at 7487.917991s 2024-03-25T18:27:13.672 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.671+0000 7fafae1db640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fafa813c480 0x7fafa814dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:57034/0 (socket says 172.21.15.3:57034) 2024-03-25T18:27:13.672 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.671+0000 7fafae1db640 1 -- 172.21.15.3:0/3523251787 learned_addr learned my addr 172.21.15.3:0/3523251787 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:27:13.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.684+0000 7fafac9d8640 1 -- 172.21.15.3:0/3523251787 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1389454637 0 0) 0x7fafa80a9f30 con 0x7fafa80b4ff0 2024-03-25T18:27:13.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.684+0000 7fafac9d8640 10 monclient(hunting): _negotiate 2024-03-25T18:27:13.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.684+0000 7fafac9d8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:13.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.684+0000 7fafac9d8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:13.686 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.684+0000 7fafac9d8640 10 monclient(hunting): my global_id is 4304 2024-03-25T18:27:13.686 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.684+0000 7fafac9d8640 1 -- 172.21.15.3:0/3523251787 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf94003520 con 0x7fafa80b4ff0 2024-03-25T18:27:13.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.687+0000 7fafac9d8640 1 -- 172.21.15.3:0/3523251787 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2764395438 0 0) 0x7fafa80ab3c0 con 0x7fafa813c480 2024-03-25T18:27:13.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.687+0000 7fafac9d8640 10 monclient(hunting): _negotiate 2024-03-25T18:27:13.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.687+0000 7fafac9d8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:13.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.687+0000 7fafac9d8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:13.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.687+0000 7fafac9d8640 10 monclient(hunting): my global_id is 4291 2024-03-25T18:27:13.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.687+0000 7fafac9d8640 1 -- 172.21.15.3:0/3523251787 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafa80a9f30 con 0x7fafa813c480 2024-03-25T18:27:13.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.693+0000 7fafac9d8640 1 -- 172.21.15.3:0/3523251787 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2115727831 0 0) 0x7faf94003520 con 0x7fafa80b4ff0 2024-03-25T18:27:13.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.693+0000 7fafac9d8640 1 -- 172.21.15.3:0/3523251787 >> v1:172.21.15.16:6789/0 conn(0x7fafa813c480 legacy=0x7fafa814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:13.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.693+0000 7fafac9d8640 1 -- 172.21.15.3:0/3523251787 >> v1:172.21.15.3:6789/0 conn(0x7fafa80b4670 legacy=0x7fafa81425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:13.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.693+0000 7fafac9d8640 10 monclient: _finish_hunting 0 2024-03-25T18:27:13.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.693+0000 7fafac9d8640 1 monclient: found mon.noname-b 2024-03-25T18:27:13.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.693+0000 7fafac9d8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:13.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.693+0000 7fafac9d8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-25T18:27:13.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.693+0000 7fafac9d8640 1 -- 172.21.15.3:0/3523251787 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa80a9b70 con 0x7fafa80b4ff0 2024-03-25T18:27:13.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.693+0000 7fafac9d8640 10 monclient: _finish_auth 0 2024-03-25T18:27:13.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.693+0000 7fafac9d8640 10 monclient: _check_auth_tickets 2024-03-25T18:27:13.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.693+0000 7fafac9d8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:13.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.693+0000 7fafac9d8640 1 -- 172.21.15.3:0/3523251787 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fafa4001690 con 0x7fafa80b4ff0 2024-03-25T18:27:13.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.693+0000 7fafafc3e640 5 monclient: authenticate success, global_id 4304 2024-03-25T18:27:13.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.693+0000 7fafafc3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:13.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.693+0000 7fafac9d8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:13.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.693+0000 7fafac9d8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:13.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.693+0000 7fafac9d8640 10 monclient: dump: 2024-03-25T18:27:13.696 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:13.697 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:13.697 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:13.697 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:13.697 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:13.697 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:13.697 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:13.697 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:13.697 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:13.697 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:13.697 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.693+0000 7fafafc3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:13.715 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.714+0000 7fafac9d8640 1 -- 172.21.15.3:0/3523251787 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafa40017f0 con 0x7fafa80b4ff0 2024-03-25T18:27:13.716 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.714+0000 7fafac9d8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:13.716 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.714+0000 7fafac9d8640 1 -- 172.21.15.3:0/3523251787 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fafa4001c00 con 0x7fafa80b4ff0 2024-03-25T18:27:13.716 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.714+0000 7fafafc3e640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:13.716 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.714+0000 7fafafc3e640 4 set_mon_vals no callback set 2024-03-25T18:27:13.716 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.714+0000 7fafac9d8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:13.716 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.714+0000 7fafac9d8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:13.716 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.714+0000 7fafac9d8640 10 monclient: dump: 2024-03-25T18:27:13.716 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:13.716 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:13.716 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:13.716 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:13.716 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:13.717 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:13.717 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:13.717 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:13.717 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:13.717 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:13.719 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.715+0000 7fafafc3e640 10 monclient: shutdown 2024-03-25T18:27:13.719 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.715+0000 7fafafc3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:13.719 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.715+0000 7fafafc3e640 1 -- 172.21.15.3:0/3523251787 >> v1:172.21.15.3:6790/0 conn(0x7fafa80b4ff0 legacy=0x7fafa813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:13.719 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.715+0000 7fafafc3e640 1 -- 172.21.15.3:0/3523251787 shutdown_connections 2024-03-25T18:27:13.719 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.715+0000 7fafafc3e640 1 -- 172.21.15.3:0/3523251787 >> 172.21.15.3:0/3523251787 conn(0x7fafa80617a0 msgr2=0x7fafa8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:13.719 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.715+0000 7fafafc3e640 1 -- 172.21.15.3:0/3523251787 shutdown_connections 2024-03-25T18:27:13.719 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.715+0000 7fafafc3e640 1 -- 172.21.15.3:0/3523251787 wait complete. 2024-03-25T18:27:13.719 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.716+0000 7fafafc3e640 10 monclient: build_initial_monmap 2024-03-25T18:27:13.719 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.716+0000 7fafafc3e640 10 monclient: monmap: 2024-03-25T18:27:13.719 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:27:13.719 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:13.720 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:27:13.720 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:27:13.720 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:13.720 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:27:13.720 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:27:13.720 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:27:13.720 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:27:13.720 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:13.720 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.716+0000 7fafafc3e640 1 librados: starting msgr at 2024-03-25T18:27:13.720 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.716+0000 7fafafc3e640 1 librados: starting objecter 2024-03-25T18:27:13.720 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.717+0000 7fafafc3e640 1 Processor -- start 2024-03-25T18:27:13.720 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.717+0000 7fafafc3e640 1 -- start start 2024-03-25T18:27:13.720 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.717+0000 7fafafc3e640 1 librados: setting wanted keys 2024-03-25T18:27:13.720 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.717+0000 7fafafc3e640 1 librados: calling monclient init 2024-03-25T18:27:13.721 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.717+0000 7fafafc3e640 10 monclient: init 2024-03-25T18:27:13.721 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.717+0000 7fafafc3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:13.721 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.717+0000 7fafafc3e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:27:13.721 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.717+0000 7fafafc3e640 10 monclient(hunting): picked mon.noname-b con 0x7fafa80b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:27:13.721 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.718+0000 7fafafc3e640 10 monclient(hunting): picked mon.noname-c con 0x7fafa80b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:27:13.721 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.718+0000 7fafafc3e640 10 monclient(hunting): picked mon.noname-a con 0x7fafa813c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:27:13.721 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.718+0000 7fafafc3e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa8144c50 con 0x7fafa813c480 2024-03-25T18:27:13.721 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.718+0000 7fafafc3e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa8144f30 con 0x7fafa80b4670 2024-03-25T18:27:13.721 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.718+0000 7fafafc3e640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa813f930 con 0x7fafa80b4ff0 2024-03-25T18:27:13.721 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.718+0000 7fafafc3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:13.721 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.718+0000 7fafafc3e640 10 monclient(hunting): authenticate will time out at 7487.978513s 2024-03-25T18:27:13.724 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.723+0000 7fafad9da640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fafa80b4670 0x7fafa81db620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:57084/0 (socket says 172.21.15.3:57084) 2024-03-25T18:27:13.724 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.723+0000 7fafad9da640 1 -- 172.21.15.3:0/3303061350 learned_addr learned my addr 172.21.15.3:0/3303061350 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:27:13.727 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.726+0000 7faf9e7fc640 10 client.?.objecter ms_handle_connect 0x7fafa80b4670 2024-03-25T18:27:13.727 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.726+0000 7faf9e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:13.734 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.733+0000 7faf9e7fc640 10 client.?.objecter ms_handle_connect 0x7fafa813c480 2024-03-25T18:27:13.734 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.733+0000 7faf9e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:13.735 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.734+0000 7faf9e7fc640 1 -- 172.21.15.3:0/3303061350 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3814716226 0 0) 0x7fafa8144f30 con 0x7fafa80b4670 2024-03-25T18:27:13.735 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.734+0000 7faf9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:13.735 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.734+0000 7faf9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:13.735 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.734+0000 7faf9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:13.735 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.734+0000 7faf9e7fc640 10 monclient(hunting): my global_id is 4294 2024-03-25T18:27:13.735 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.734+0000 7faf9e7fc640 1 -- 172.21.15.3:0/3303061350 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf8c004eb0 con 0x7fafa80b4670 2024-03-25T18:27:13.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.742+0000 7faf9e7fc640 10 client.?.objecter ms_handle_connect 0x7fafa80b4ff0 2024-03-25T18:27:13.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.742+0000 7faf9e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:13.748 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.747+0000 7faf9e7fc640 1 -- 172.21.15.3:0/3303061350 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2921351266 0 0) 0x7fafa8144c50 con 0x7fafa813c480 2024-03-25T18:27:13.748 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.747+0000 7faf9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:13.748 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.747+0000 7faf9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:13.749 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.747+0000 7faf9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:13.749 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.747+0000 7faf9e7fc640 10 monclient(hunting): my global_id is 4335 2024-03-25T18:27:13.749 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.747+0000 7faf9e7fc640 1 -- 172.21.15.3:0/3303061350 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafa8144f30 con 0x7fafa813c480 2024-03-25T18:27:13.756 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.755+0000 7faf9e7fc640 1 -- 172.21.15.3:0/3303061350 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2475335603 0 0) 0x7fafa813f930 con 0x7fafa80b4ff0 2024-03-25T18:27:13.756 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.755+0000 7faf9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:13.756 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.755+0000 7faf9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:13.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.755+0000 7faf9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:13.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.755+0000 7faf9e7fc640 10 monclient(hunting): my global_id is 4307 2024-03-25T18:27:13.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.755+0000 7faf9e7fc640 1 -- 172.21.15.3:0/3303061350 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafa8144c50 con 0x7fafa80b4ff0 2024-03-25T18:27:13.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.768+0000 7faf9e7fc640 1 -- 172.21.15.3:0/3303061350 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2463550996 0 0) 0x7faf8c004eb0 con 0x7fafa80b4670 2024-03-25T18:27:13.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.768+0000 7faf9e7fc640 1 -- 172.21.15.3:0/3303061350 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafa813f930 con 0x7fafa80b4670 2024-03-25T18:27:13.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.768+0000 7faf9e7fc640 1 -- 172.21.15.3:0/3303061350 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fafa4001bc0 con 0x7fafa80b4670 2024-03-25T18:27:13.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.768+0000 7faf9e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:13.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.768+0000 7faf9e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:13.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.768+0000 7faf9e7fc640 10 monclient(hunting): dump: 2024-03-25T18:27:13.770 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:13.770 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:13.770 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:13.770 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:13.770 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:13.770 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:13.770 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:13.770 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:13.770 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:13.770 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:13.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.768+0000 7faf9e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:13.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.774+0000 7faf9e7fc640 1 -- 172.21.15.3:0/3303061350 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2429579102 0 0) 0x7fafa8144f30 con 0x7fafa813c480 2024-03-25T18:27:13.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.774+0000 7faf9e7fc640 1 -- 172.21.15.3:0/3303061350 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf8c004eb0 con 0x7fafa813c480 2024-03-25T18:27:13.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.774+0000 7faf9e7fc640 1 -- 172.21.15.3:0/3303061350 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fafa0001b50 con 0x7fafa813c480 2024-03-25T18:27:13.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.774+0000 7faf9e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:13.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.774+0000 7faf9e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:13.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.774+0000 7faf9e7fc640 10 monclient(hunting): dump: 2024-03-25T18:27:13.777 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:13.777 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:13.777 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:13.777 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:13.777 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:13.777 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:13.778 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:13.778 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:13.778 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:13.778 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:13.778 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.777+0000 7faf9e7fc640 1 -- 172.21.15.3:0/3303061350 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1754155968 0 0) 0x7fafa8144c50 con 0x7fafa80b4ff0 2024-03-25T18:27:13.778 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.777+0000 7faf9e7fc640 1 -- 172.21.15.3:0/3303061350 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafa8144f30 con 0x7fafa80b4ff0 2024-03-25T18:27:13.779 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.777+0000 7faf9e7fc640 1 -- 172.21.15.3:0/3303061350 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7faf98001830 con 0x7fafa80b4ff0 2024-03-25T18:27:13.779 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.777+0000 7faf9e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:13.779 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.777+0000 7faf9e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:13.779 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.777+0000 7faf9e7fc640 10 monclient(hunting): dump: 2024-03-25T18:27:13.779 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:13.779 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:13.779 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:13.779 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:13.779 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:13.779 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:13.779 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:13.779 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:13.779 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:13.779 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:13.781 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.779+0000 7faf9e7fc640 1 -- 172.21.15.3:0/3303061350 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2913204981 0 0) 0x7fafa813f930 con 0x7fafa80b4670 2024-03-25T18:27:13.781 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.779+0000 7faf9e7fc640 1 -- 172.21.15.3:0/3303061350 >> v1:172.21.15.3:6790/0 conn(0x7fafa80b4ff0 legacy=0x7fafa81dbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:13.781 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.779+0000 7faf9e7fc640 1 -- 172.21.15.3:0/3303061350 >> v1:172.21.15.3:6789/0 conn(0x7fafa813c480 legacy=0x7fafa81e2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:13.781 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.779+0000 7faf9e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:27:13.781 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.779+0000 7faf9e7fc640 1 monclient: found mon.b 2024-03-25T18:27:13.781 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.779+0000 7faf9e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:13.781 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.779+0000 7faf9e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:27:13.782 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.779+0000 7faf9e7fc640 1 -- 172.21.15.3:0/3303061350 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa813c930 con 0x7fafa80b4670 2024-03-25T18:27:13.782 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.779+0000 7faf9e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:27:13.782 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.779+0000 7faf9e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:27:13.782 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.779+0000 7faf9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:13.782 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.779+0000 7fafafc3e640 5 monclient: authenticate success, global_id 4294 2024-03-25T18:27:13.782 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.779+0000 7fafafc3e640 10 monclient: _renew_subs 2024-03-25T18:27:13.782 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.779+0000 7fafafc3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:27:13.782 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.779+0000 7fafafc3e640 1 -- 172.21.15.3:0/3303061350 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa8146680 con 0x7fafa80b4670 2024-03-25T18:27:13.782 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.779+0000 7fafafc3e640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:13.782 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.779+0000 7fafafc3e640 10 monclient: _renew_subs 2024-03-25T18:27:13.782 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.779+0000 7fafafc3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:27:13.783 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.779+0000 7fafafc3e640 1 -- 172.21.15.3:0/3303061350 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa813f090 con 0x7fafa80b4670 2024-03-25T18:27:13.783 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.779+0000 7fafafc3e640 1 librados: init done 2024-03-25T18:27:13.783 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.780+0000 7fafafc3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:13.783 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.780+0000 7fafafc3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:13.783 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.780+0000 7fafafc3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:27:13.783 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:13.780+0000 7fafafc3e640 1 -- 172.21.15.3:0/3303061350 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa81438d0 con 0x7fafa80b4670 2024-03-25T18:27:14.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.076+0000 7faf9e7fc640 1 -- 172.21.15.3:0/3303061350 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafa40011f0 con 0x7fafa80b4670 2024-03-25T18:27:14.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.076+0000 7faf9e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:14.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.076+0000 7faf9e7fc640 1 -- 172.21.15.3:0/3303061350 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fafa4003b00 con 0x7fafa80b4670 2024-03-25T18:27:14.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.076+0000 7faf9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:14.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.076+0000 7faf9e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:14.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.076+0000 7faf9e7fc640 10 monclient: dump: 2024-03-25T18:27:14.078 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:14.078 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:14.078 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:14.078 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:14.078 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:14.078 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:14.078 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:14.078 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:14.078 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:14.078 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:14.078 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.076+0000 7faf9ffff640 4 set_mon_vals no callback set 2024-03-25T18:27:14.079 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.077+0000 7faf9e7fc640 1 -- 172.21.15.3:0/3303061350 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7fafa4003d20 con 0x7fafa80b4670 2024-03-25T18:27:14.079 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.077+0000 7fafad1d9640 10 monclient: get_auth_request con 0x7faf8c06e0c0 auth_method 0 2024-03-25T18:27:14.079 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.077+0000 7faf9e7fc640 10 client.4294.objecter ms_handle_connect 0x7faf8c06e0c0 2024-03-25T18:27:14.087 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.086+0000 7faf9e7fc640 1 -- 172.21.15.3:0/3303061350 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7fafa40548c0 con 0x7fafa80b4670 2024-03-25T18:27:14.087 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.086+0000 7faf9e7fc640 10 client.4294.objecter ms_dispatch 0x7fafa8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:14.087 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.086+0000 7faf9e7fc640 3 client.4294.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:14.087 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.086+0000 7faf9e7fc640 3 client.4294.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:14.087 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.086+0000 7faf9e7fc640 20 client.4294.objecter dump_active .. 0 homeless 2024-03-25T18:27:14.088 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.086+0000 7faf9e7fc640 1 -- 172.21.15.3:0/3303061350 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fafa40503d0 con 0x7fafa80b4670 2024-03-25T18:27:14.088 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.086+0000 7faf9e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:14.088 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.086+0000 7faf9e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:14.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.307+0000 7fafafc3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:14.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.307+0000 7fafafc3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:14.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.307+0000 7fafafc3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:27:14.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.307+0000 7fafafc3e640 1 -- 172.21.15.3:0/3303061350 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fafa8146810 con 0x7fafa80b4670 2024-03-25T18:27:14.402 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.400+0000 7faf9e7fc640 1 -- 172.21.15.3:0/3303061350 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7fafa8146810 con 0x7fafa80b4670 2024-03-25T18:27:14.402 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.400+0000 7faf9e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:14.402 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.400+0000 7faf9e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:14.402 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:27:14.403 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:14.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.403+0000 7fafafc3e640 10 librados: watch_flush enter 2024-03-25T18:27:14.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.403+0000 7fafafc3e640 10 librados: watch_flush exit 2024-03-25T18:27:14.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.403+0000 7fafafc3e640 20 client.4294.objecter shutdown clearing up homeless session... 2024-03-25T18:27:14.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.403+0000 7fafafc3e640 10 client.4294.objecter successfully canceled tick 2024-03-25T18:27:14.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.403+0000 7fafafc3e640 1 -- 172.21.15.3:0/3303061350 >> v1:172.21.15.3:6800/4195140966 conn(0x7faf8c06e0c0 legacy=0x7faf8c070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:14.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.403+0000 7fafafc3e640 10 monclient: shutdown 2024-03-25T18:27:14.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.403+0000 7fafafc3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:14.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.403+0000 7fafafc3e640 1 -- 172.21.15.3:0/3303061350 >> v1:172.21.15.16:6789/0 conn(0x7fafa80b4670 legacy=0x7fafa81db620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:14.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.403+0000 7fafafc3e640 1 -- 172.21.15.3:0/3303061350 shutdown_connections 2024-03-25T18:27:14.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.403+0000 7fafafc3e640 1 -- 172.21.15.3:0/3303061350 >> 172.21.15.3:0/3303061350 conn(0x7fafa80617a0 msgr2=0x7fafa8140670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:14.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.404+0000 7fafafc3e640 1 -- 172.21.15.3:0/3303061350 shutdown_connections 2024-03-25T18:27:14.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.404+0000 7fafafc3e640 1 -- 172.21.15.3:0/3303061350 wait complete. 2024-03-25T18:27:14.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:14.404+0000 7fafafc3e640 1 librados: shutdown 2024-03-25T18:27:14.416 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:21.060 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:02:34.789 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:21.110 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:02:34.839 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:21.138 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:02:34.866 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:21.163 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:02:34.892 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:21.168 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:02:34.898 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:21.192 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:02:34.884 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:21.200 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:02:34.892 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:21.207 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:02:34.936 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:21.221 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:02:34.914 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:21.223 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:21.337 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.335+0000 7f6c08f15640 10 monclient: get_monmap_and_config 2024-03-25T18:27:21.337 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.335+0000 7f6c08f15640 10 monclient: build_initial_monmap 2024-03-25T18:27:21.337 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.335+0000 7f6c08f15640 10 monclient: monmap: 2024-03-25T18:27:21.337 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:27:21.337 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:21.337 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:27:21.337004+0000 2024-03-25T18:27:21.337 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:27:21.337004+0000 2024-03-25T18:27:21.337 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:21.337 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:21.337 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:27:21.337 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:27:21.337 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:27:21.337 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:21.339 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.336+0000 7f6c08f15640 1 Processor -- start 2024-03-25T18:27:21.342 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.336+0000 7f6c08f15640 1 -- start start 2024-03-25T18:27:21.342 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.336+0000 7f6c08f15640 10 monclient: init 2024-03-25T18:27:21.342 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.337+0000 7f6c08f15640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:21.342 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.337+0000 7f6c08f15640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:27:21.342 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.337+0000 7f6c08f15640 10 monclient(hunting): picked mon.noname-c con 0x7f6c040b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:27:21.342 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.337+0000 7f6c08f15640 10 monclient(hunting): picked mon.noname-b con 0x7f6c040b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:27:21.342 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.337+0000 7f6c08f15640 10 monclient(hunting): picked mon.noname-a con 0x7f6c0413c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:27:21.343 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.337+0000 7f6c08f15640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c040ab0e0 con 0x7f6c0413c480 2024-03-25T18:27:21.343 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.337+0000 7f6c08f15640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c040ab3c0 con 0x7f6c040b4ff0 2024-03-25T18:27:21.343 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.337+0000 7f6c08f15640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c040a9f30 con 0x7f6c040b4670 2024-03-25T18:27:21.343 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.337+0000 7f6c08f15640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:21.343 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.337+0000 7f6c08f15640 10 monclient(hunting): authenticate will time out at 7495.597906s 2024-03-25T18:27:21.348 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.347+0000 7f6c02d9d640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f6c0413c480 0x7f6c0414dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50318/0 (socket says 172.21.15.3:50318) 2024-03-25T18:27:21.348 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.347+0000 7f6c02d9d640 1 -- 172.21.15.3:0/716416360 learned_addr learned my addr 172.21.15.3:0/716416360 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:27:21.362 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.361+0000 7f6c0159a640 1 -- 172.21.15.3:0/716416360 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1011447958 0 0) 0x7f6c040ab0e0 con 0x7f6c0413c480 2024-03-25T18:27:21.362 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.361+0000 7f6c0159a640 10 monclient(hunting): _negotiate 2024-03-25T18:27:21.362 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.361+0000 7f6c0159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:21.362 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.361+0000 7f6c0159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:21.362 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.361+0000 7f6c0159a640 10 monclient(hunting): my global_id is 4338 2024-03-25T18:27:21.362 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.361+0000 7f6c0159a640 1 -- 172.21.15.3:0/716416360 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bec003520 con 0x7f6c0413c480 2024-03-25T18:27:21.372 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.371+0000 7f6c0159a640 1 -- 172.21.15.3:0/716416360 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3487124589 0 0) 0x7f6c040a9f30 con 0x7f6c040b4670 2024-03-25T18:27:21.372 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.371+0000 7f6c0159a640 10 monclient(hunting): _negotiate 2024-03-25T18:27:21.372 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.371+0000 7f6c0159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:21.372 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.371+0000 7f6c0159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:21.372 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.371+0000 7f6c0159a640 10 monclient(hunting): my global_id is 4310 2024-03-25T18:27:21.372 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.371+0000 7f6c0159a640 1 -- 172.21.15.3:0/716416360 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c040ab0e0 con 0x7f6c040b4670 2024-03-25T18:27:21.379 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.378+0000 7f6c0159a640 1 -- 172.21.15.3:0/716416360 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 438785207 0 0) 0x7f6bec003520 con 0x7f6c0413c480 2024-03-25T18:27:21.379 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.378+0000 7f6c0159a640 1 -- 172.21.15.3:0/716416360 >> v1:172.21.15.3:6790/0 conn(0x7f6c040b4670 legacy=0x7f6c041425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:21.379 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.378+0000 7f6c0159a640 1 -- 172.21.15.3:0/716416360 >> v1:172.21.15.16:6789/0 conn(0x7f6c040b4ff0 legacy=0x7f6c0413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:21.379 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.378+0000 7f6c0159a640 10 monclient: _finish_hunting 0 2024-03-25T18:27:21.379 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.378+0000 7f6c0159a640 1 monclient: found mon.noname-a 2024-03-25T18:27:21.379 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.378+0000 7f6c0159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:21.379 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.378+0000 7f6c0159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:27:21.379 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.378+0000 7f6c0159a640 1 -- 172.21.15.3:0/716416360 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c040a9b70 con 0x7f6c0413c480 2024-03-25T18:27:21.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.378+0000 7f6c0159a640 10 monclient: _finish_auth 0 2024-03-25T18:27:21.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.378+0000 7f6c0159a640 10 monclient: _check_auth_tickets 2024-03-25T18:27:21.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.378+0000 7f6c0159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:21.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.378+0000 7f6c0159a640 1 -- 172.21.15.3:0/716416360 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f6bfc001a10 con 0x7f6c0413c480 2024-03-25T18:27:21.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.378+0000 7f6c0159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:21.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.378+0000 7f6c0159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:21.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.378+0000 7f6c0159a640 10 monclient: dump: 2024-03-25T18:27:21.380 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:21.380 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:21.380 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:21.380 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:21.380 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:21.380 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:21.380 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:21.380 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:21.380 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:21.381 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:21.381 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.378+0000 7f6c08f15640 5 monclient: authenticate success, global_id 4338 2024-03-25T18:27:21.381 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.378+0000 7f6c08f15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:21.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.386+0000 7f6c0159a640 1 -- 172.21.15.3:0/716416360 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6bfc001b70 con 0x7f6c0413c480 2024-03-25T18:27:21.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.386+0000 7f6c0159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:21.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.386+0000 7f6c0159a640 1 -- 172.21.15.3:0/716416360 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f6bfc001f80 con 0x7f6c0413c480 2024-03-25T18:27:21.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.386+0000 7f6c0159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:21.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.386+0000 7f6c0159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:21.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.386+0000 7f6c0159a640 10 monclient: dump: 2024-03-25T18:27:21.387 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:21.387 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:21.388 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:21.388 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:21.388 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:21.388 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:21.388 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:21.388 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:21.388 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:21.388 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:21.388 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.386+0000 7f6c08f15640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:21.388 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.386+0000 7f6c08f15640 4 set_mon_vals no callback set 2024-03-25T18:27:21.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.387+0000 7f6c08f15640 10 monclient: shutdown 2024-03-25T18:27:21.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.387+0000 7f6c08f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:21.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.387+0000 7f6c08f15640 1 -- 172.21.15.3:0/716416360 >> v1:172.21.15.3:6789/0 conn(0x7f6c0413c480 legacy=0x7f6c0414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:21.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.387+0000 7f6c08f15640 1 -- 172.21.15.3:0/716416360 shutdown_connections 2024-03-25T18:27:21.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.387+0000 7f6c08f15640 1 -- 172.21.15.3:0/716416360 >> 172.21.15.3:0/716416360 conn(0x7f6c040617a0 msgr2=0x7f6c04061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:21.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.387+0000 7f6c08f15640 1 -- 172.21.15.3:0/716416360 shutdown_connections 2024-03-25T18:27:21.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.387+0000 7f6c08f15640 1 -- 172.21.15.3:0/716416360 wait complete. 2024-03-25T18:27:21.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.388+0000 7f6c08f15640 10 monclient: build_initial_monmap 2024-03-25T18:27:21.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.388+0000 7f6c08f15640 10 monclient: monmap: 2024-03-25T18:27:21.390 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:27:21.390 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:21.390 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:27:21.390 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:27:21.390 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:21.391 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:27:21.391 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:27:21.391 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:27:21.391 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:27:21.391 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:21.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.388+0000 7f6c08f15640 1 librados: starting msgr at 2024-03-25T18:27:21.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.388+0000 7f6c08f15640 1 librados: starting objecter 2024-03-25T18:27:21.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.388+0000 7f6c08f15640 1 Processor -- start 2024-03-25T18:27:21.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.388+0000 7f6c08f15640 1 -- start start 2024-03-25T18:27:21.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.388+0000 7f6c08f15640 1 librados: setting wanted keys 2024-03-25T18:27:21.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.388+0000 7f6c08f15640 1 librados: calling monclient init 2024-03-25T18:27:21.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.388+0000 7f6c08f15640 10 monclient: init 2024-03-25T18:27:21.392 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.389+0000 7f6c08f15640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:21.392 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.389+0000 7f6c08f15640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:27:21.392 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.389+0000 7f6c08f15640 10 monclient(hunting): picked mon.noname-a con 0x7f6c040b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:27:21.392 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.389+0000 7f6c08f15640 10 monclient(hunting): picked mon.noname-c con 0x7f6c040b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:27:21.392 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.389+0000 7f6c08f15640 10 monclient(hunting): picked mon.noname-b con 0x7f6c0413c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:27:21.392 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.389+0000 7f6c08f15640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c040aefb0 con 0x7f6c040b4670 2024-03-25T18:27:21.393 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.389+0000 7f6c08f15640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c040b0220 con 0x7f6c0413c480 2024-03-25T18:27:21.393 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.389+0000 7f6c08f15640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c040b0500 con 0x7f6c040b4ff0 2024-03-25T18:27:21.393 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.389+0000 7f6c08f15640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:21.393 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.389+0000 7f6c08f15640 10 monclient(hunting): authenticate will time out at 7495.649957s 2024-03-25T18:27:21.394 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.393+0000 7f6c01d9b640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f6c040b4ff0 0x7f6c040b0c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:54848/0 (socket says 172.21.15.3:54848) 2024-03-25T18:27:21.394 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.393+0000 7f6c01d9b640 1 -- 172.21.15.3:0/2125925034 learned_addr learned my addr 172.21.15.3:0/2125925034 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:27:21.399 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.398+0000 7f6bf2ffd640 10 client.?.objecter ms_handle_connect 0x7f6c040b4ff0 2024-03-25T18:27:21.400 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.398+0000 7f6bf2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:21.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.404+0000 7f6bf2ffd640 10 client.?.objecter ms_handle_connect 0x7f6c040b4670 2024-03-25T18:27:21.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.404+0000 7f6bf2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:21.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.408+0000 7f6bf2ffd640 10 client.?.objecter ms_handle_connect 0x7f6c0413c480 2024-03-25T18:27:21.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.408+0000 7f6bf2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:21.417 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.416+0000 7f6bf2ffd640 1 -- 172.21.15.3:0/2125925034 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2239578281 0 0) 0x7f6c040b0500 con 0x7f6c040b4ff0 2024-03-25T18:27:21.418 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.416+0000 7f6bf2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:21.418 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.416+0000 7f6bf2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:21.418 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.416+0000 7f6bf2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:21.418 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.416+0000 7f6bf2ffd640 10 monclient(hunting): my global_id is 4313 2024-03-25T18:27:21.418 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.416+0000 7f6bf2ffd640 1 -- 172.21.15.3:0/2125925034 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6be4004eb0 con 0x7f6c040b4ff0 2024-03-25T18:27:21.419 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.417+0000 7f6bf2ffd640 1 -- 172.21.15.3:0/2125925034 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2574113776 0 0) 0x7f6c040b0220 con 0x7f6c0413c480 2024-03-25T18:27:21.419 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.417+0000 7f6bf2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:21.419 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.417+0000 7f6bf2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:21.419 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.417+0000 7f6bf2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:21.419 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.417+0000 7f6bf2ffd640 10 monclient(hunting): my global_id is 4300 2024-03-25T18:27:21.419 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.417+0000 7f6bf2ffd640 1 -- 172.21.15.3:0/2125925034 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c040b0500 con 0x7f6c0413c480 2024-03-25T18:27:21.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.423+0000 7f6bf2ffd640 1 -- 172.21.15.3:0/2125925034 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3879472958 0 0) 0x7f6c040aefb0 con 0x7f6c040b4670 2024-03-25T18:27:21.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.423+0000 7f6bf2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:21.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.423+0000 7f6bf2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:21.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.423+0000 7f6bf2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:21.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.423+0000 7f6bf2ffd640 10 monclient(hunting): my global_id is 4341 2024-03-25T18:27:21.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.423+0000 7f6bf2ffd640 1 -- 172.21.15.3:0/2125925034 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c040b0220 con 0x7f6c040b4670 2024-03-25T18:27:21.431 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.430+0000 7f6bf2ffd640 1 -- 172.21.15.3:0/2125925034 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1207518097 0 0) 0x7f6be4004eb0 con 0x7f6c040b4ff0 2024-03-25T18:27:21.431 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.430+0000 7f6bf2ffd640 1 -- 172.21.15.3:0/2125925034 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c040aefb0 con 0x7f6c040b4ff0 2024-03-25T18:27:21.431 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.430+0000 7f6bf2ffd640 1 -- 172.21.15.3:0/2125925034 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f6bf8001a10 con 0x7f6c040b4ff0 2024-03-25T18:27:21.431 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.430+0000 7f6bf2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:21.431 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.430+0000 7f6bf2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:21.431 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.430+0000 7f6bf2ffd640 10 monclient(hunting): dump: 2024-03-25T18:27:21.432 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:21.432 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:21.432 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:21.432 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:21.432 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:21.432 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:21.432 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:21.432 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:21.432 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:21.432 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:21.432 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.430+0000 7f6bf2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:21.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.443+0000 7f6bf2ffd640 1 -- 172.21.15.3:0/2125925034 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4270834748 0 0) 0x7f6c040b0500 con 0x7f6c0413c480 2024-03-25T18:27:21.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.443+0000 7f6bf2ffd640 1 -- 172.21.15.3:0/2125925034 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6be4004eb0 con 0x7f6c0413c480 2024-03-25T18:27:21.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.443+0000 7f6bf2ffd640 1 -- 172.21.15.3:0/2125925034 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f6bfc001e80 con 0x7f6c0413c480 2024-03-25T18:27:21.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.443+0000 7f6bf2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:21.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.443+0000 7f6bf2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:21.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.443+0000 7f6bf2ffd640 10 monclient(hunting): dump: 2024-03-25T18:27:21.445 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:21.445 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:21.445 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:21.445 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:21.445 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:21.445 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:21.445 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:21.445 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:21.445 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:21.445 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:21.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.447+0000 7f6bf2ffd640 1 -- 172.21.15.3:0/2125925034 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3905640646 0 0) 0x7f6c040aefb0 con 0x7f6c040b4ff0 2024-03-25T18:27:21.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.447+0000 7f6bf2ffd640 1 -- 172.21.15.3:0/2125925034 >> v1:172.21.15.16:6789/0 conn(0x7f6c0413c480 legacy=0x7f6c041e2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:21.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.447+0000 7f6bf2ffd640 1 -- 172.21.15.3:0/2125925034 >> v1:172.21.15.3:6789/0 conn(0x7f6c040b4670 legacy=0x7f6c040ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:21.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.447+0000 7f6bf2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:27:21.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.447+0000 7f6bf2ffd640 1 monclient: found mon.c 2024-03-25T18:27:21.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.447+0000 7f6bf2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:21.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.447+0000 7f6bf2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:27:21.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.447+0000 7f6bf2ffd640 1 -- 172.21.15.3:0/2125925034 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c040b07e0 con 0x7f6c040b4ff0 2024-03-25T18:27:21.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.447+0000 7f6bf2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:27:21.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.447+0000 7f6bf2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:27:21.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.447+0000 7f6bf2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:21.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.447+0000 7f6c08f15640 5 monclient: authenticate success, global_id 4313 2024-03-25T18:27:21.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.447+0000 7f6c08f15640 10 monclient: _renew_subs 2024-03-25T18:27:21.450 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.447+0000 7f6c08f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:27:21.450 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.447+0000 7f6c08f15640 1 -- 172.21.15.3:0/2125925034 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c040aebb0 con 0x7f6c040b4ff0 2024-03-25T18:27:21.450 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.447+0000 7f6c08f15640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:21.450 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.447+0000 7f6c08f15640 10 monclient: _renew_subs 2024-03-25T18:27:21.450 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.447+0000 7f6c08f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:27:21.450 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.447+0000 7f6c08f15640 1 -- 172.21.15.3:0/2125925034 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c040b14f0 con 0x7f6c040b4ff0 2024-03-25T18:27:21.450 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.447+0000 7f6c08f15640 1 librados: init done 2024-03-25T18:27:21.450 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.448+0000 7f6c08f15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:21.451 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.448+0000 7f6c08f15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:21.451 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.448+0000 7f6c08f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:27:21.451 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.448+0000 7f6c08f15640 1 -- 172.21.15.3:0/2125925034 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c0406dbd0 con 0x7f6c040b4ff0 2024-03-25T18:27:21.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.772+0000 7f6bf2ffd640 1 -- 172.21.15.3:0/2125925034 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6bf8001c50 con 0x7f6c040b4ff0 2024-03-25T18:27:21.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.772+0000 7f6bf2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:21.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.772+0000 7f6bf2ffd640 1 -- 172.21.15.3:0/2125925034 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f6bf8003a60 con 0x7f6c040b4ff0 2024-03-25T18:27:21.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.772+0000 7f6bf2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:21.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.772+0000 7f6bf2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:21.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.772+0000 7f6bf2ffd640 10 monclient: dump: 2024-03-25T18:27:21.773 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:21.774 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:21.774 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:21.774 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:21.774 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:21.774 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:21.774 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:21.774 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:21.774 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:21.774 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:21.774 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.772+0000 7f6c00d99640 4 set_mon_vals no callback set 2024-03-25T18:27:21.775 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.772+0000 7f6bf2ffd640 1 -- 172.21.15.3:0/2125925034 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f6bf8003c80 con 0x7f6c040b4ff0 2024-03-25T18:27:21.775 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.773+0000 7f6c0259c640 10 monclient: get_auth_request con 0x7f6be406df70 auth_method 0 2024-03-25T18:27:21.775 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.773+0000 7f6bf2ffd640 10 client.4313.objecter ms_handle_connect 0x7f6be406df70 2024-03-25T18:27:21.780 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.779+0000 7f6bf2ffd640 1 -- 172.21.15.3:0/2125925034 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f6bf8054830 con 0x7f6c040b4ff0 2024-03-25T18:27:21.780 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.779+0000 7f6bf2ffd640 10 client.4313.objecter ms_dispatch 0x7f6c04000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:21.780 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.779+0000 7f6bf2ffd640 3 client.4313.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:21.780 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.779+0000 7f6bf2ffd640 3 client.4313.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:21.780 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.779+0000 7f6bf2ffd640 20 client.4313.objecter dump_active .. 0 homeless 2024-03-25T18:27:21.781 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.779+0000 7f6bf2ffd640 1 -- 172.21.15.3:0/2125925034 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6bf8050340 con 0x7f6c040b4ff0 2024-03-25T18:27:21.781 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.779+0000 7f6bf2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:21.781 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:21.779+0000 7f6bf2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:22.030 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:22.028+0000 7f6c08f15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:22.030 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:22.028+0000 7f6c08f15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:22.030 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:22.028+0000 7f6c08f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:27:22.030 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:22.028+0000 7f6c08f15640 1 -- 172.21.15.3:0/2125925034 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c040aed40 con 0x7f6c040b4ff0 2024-03-25T18:27:22.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:22.067+0000 7f6bf2ffd640 1 -- 172.21.15.3:0/2125925034 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f6bf8052270 con 0x7f6c040b4ff0 2024-03-25T18:27:22.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:22.067+0000 7f6bf2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:22.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:22.067+0000 7f6bf2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:22.069 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:27:22.069 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:22.070 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:22.069+0000 7f6c08f15640 10 librados: watch_flush enter 2024-03-25T18:27:22.070 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:22.069+0000 7f6c08f15640 10 librados: watch_flush exit 2024-03-25T18:27:22.070 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:22.069+0000 7f6c08f15640 20 client.4313.objecter shutdown clearing up homeless session... 2024-03-25T18:27:22.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:22.069+0000 7f6c08f15640 10 client.4313.objecter successfully canceled tick 2024-03-25T18:27:22.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:22.069+0000 7f6c08f15640 1 -- 172.21.15.3:0/2125925034 >> v1:172.21.15.3:6800/4195140966 conn(0x7f6be406df70 legacy=0x7f6be4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:22.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:22.069+0000 7f6c08f15640 10 monclient: shutdown 2024-03-25T18:27:22.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:22.069+0000 7f6c08f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:22.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:22.069+0000 7f6c08f15640 1 -- 172.21.15.3:0/2125925034 >> v1:172.21.15.3:6790/0 conn(0x7f6c040b4ff0 legacy=0x7f6c040b0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:22.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:22.069+0000 7f6c08f15640 1 -- 172.21.15.3:0/2125925034 shutdown_connections 2024-03-25T18:27:22.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:22.069+0000 7f6c08f15640 1 -- 172.21.15.3:0/2125925034 >> 172.21.15.3:0/2125925034 conn(0x7f6c040617a0 msgr2=0x7f6c040b2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:22.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:22.069+0000 7f6c08f15640 1 -- 172.21.15.3:0/2125925034 shutdown_connections 2024-03-25T18:27:22.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:22.069+0000 7f6c08f15640 1 -- 172.21.15.3:0/2125925034 wait complete. 2024-03-25T18:27:22.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:22.069+0000 7f6c08f15640 1 librados: shutdown 2024-03-25T18:27:22.082 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:28.889 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:29.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.004+0000 7ff050d0e640 10 monclient: get_monmap_and_config 2024-03-25T18:27:29.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.004+0000 7ff050d0e640 10 monclient: build_initial_monmap 2024-03-25T18:27:29.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.004+0000 7ff050d0e640 10 monclient: monmap: 2024-03-25T18:27:29.005 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:27:29.006 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:29.006 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:27:29.005436+0000 2024-03-25T18:27:29.006 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:27:29.005436+0000 2024-03-25T18:27:29.006 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:29.006 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:29.006 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:27:29.006 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:27:29.006 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:27:29.006 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:29.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.004+0000 7ff050d0e640 1 Processor -- start 2024-03-25T18:27:29.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.005+0000 7ff050d0e640 1 -- start start 2024-03-25T18:27:29.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.005+0000 7ff050d0e640 10 monclient: init 2024-03-25T18:27:29.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.006+0000 7ff050d0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:29.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.006+0000 7ff050d0e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:27:29.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.006+0000 7ff050d0e640 10 monclient(hunting): picked mon.noname-b con 0x7ff04c0b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:27:29.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.006+0000 7ff050d0e640 10 monclient(hunting): picked mon.noname-a con 0x7ff04c0b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:27:29.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.006+0000 7ff050d0e640 10 monclient(hunting): picked mon.noname-c con 0x7ff04c13c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:27:29.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.006+0000 7ff050d0e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff04c0ab0e0 con 0x7ff04c0b4670 2024-03-25T18:27:29.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.006+0000 7ff050d0e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff04c0ab3c0 con 0x7ff04c13c480 2024-03-25T18:27:29.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.006+0000 7ff050d0e640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff04c0a9f30 con 0x7ff04c0b4ff0 2024-03-25T18:27:29.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.006+0000 7ff050d0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:29.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.006+0000 7ff050d0e640 10 monclient(hunting): authenticate will time out at 7503.266591s 2024-03-25T18:27:29.013 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.012+0000 7ff04a59c640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7ff04c0b4ff0 0x7ff04c13bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:35858/0 (socket says 172.21.15.3:35858) 2024-03-25T18:27:29.013 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.012+0000 7ff04a59c640 1 -- 172.21.15.3:0/4145898774 learned_addr learned my addr 172.21.15.3:0/4145898774 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:27:29.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.031+0000 7ff04959a640 1 -- 172.21.15.3:0/4145898774 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2004998015 0 0) 0x7ff04c0ab3c0 con 0x7ff04c13c480 2024-03-25T18:27:29.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.031+0000 7ff04959a640 10 monclient(hunting): _negotiate 2024-03-25T18:27:29.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.031+0000 7ff04959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:29.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.031+0000 7ff04959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:29.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.031+0000 7ff04959a640 10 monclient(hunting): my global_id is 4303 2024-03-25T18:27:29.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.031+0000 7ff04959a640 1 -- 172.21.15.3:0/4145898774 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff034003520 con 0x7ff04c13c480 2024-03-25T18:27:29.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.031+0000 7ff04959a640 1 -- 172.21.15.3:0/4145898774 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2481208257 0 0) 0x7ff04c0a9f30 con 0x7ff04c0b4ff0 2024-03-25T18:27:29.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.031+0000 7ff04959a640 10 monclient(hunting): _negotiate 2024-03-25T18:27:29.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.031+0000 7ff04959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:29.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.031+0000 7ff04959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:29.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.031+0000 7ff04959a640 10 monclient(hunting): my global_id is 4316 2024-03-25T18:27:29.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.031+0000 7ff04959a640 1 -- 172.21.15.3:0/4145898774 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff04c0ab3c0 con 0x7ff04c0b4ff0 2024-03-25T18:27:29.035 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.034+0000 7ff04959a640 1 -- 172.21.15.3:0/4145898774 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1965156950 0 0) 0x7ff04c0ab0e0 con 0x7ff04c0b4670 2024-03-25T18:27:29.035 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.034+0000 7ff04959a640 10 monclient(hunting): _negotiate 2024-03-25T18:27:29.035 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.034+0000 7ff04959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:29.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.034+0000 7ff04959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:29.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.034+0000 7ff04959a640 10 monclient(hunting): my global_id is 4344 2024-03-25T18:27:29.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.034+0000 7ff04959a640 1 -- 172.21.15.3:0/4145898774 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff04c0a9f30 con 0x7ff04c0b4670 2024-03-25T18:27:29.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.041+0000 7ff04959a640 1 -- 172.21.15.3:0/4145898774 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1888309118 0 0) 0x7ff04c0ab3c0 con 0x7ff04c0b4ff0 2024-03-25T18:27:29.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.041+0000 7ff04959a640 1 -- 172.21.15.3:0/4145898774 >> v1:172.21.15.16:6789/0 conn(0x7ff04c13c480 legacy=0x7ff04c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:29.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.041+0000 7ff04959a640 1 -- 172.21.15.3:0/4145898774 >> v1:172.21.15.3:6789/0 conn(0x7ff04c0b4670 legacy=0x7ff04c1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:29.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.041+0000 7ff04959a640 10 monclient: _finish_hunting 0 2024-03-25T18:27:29.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.041+0000 7ff04959a640 1 monclient: found mon.noname-b 2024-03-25T18:27:29.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.041+0000 7ff04959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:29.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.041+0000 7ff04959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-25T18:27:29.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.041+0000 7ff04959a640 1 -- 172.21.15.3:0/4145898774 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff04c0a9b70 con 0x7ff04c0b4ff0 2024-03-25T18:27:29.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.041+0000 7ff04959a640 10 monclient: _finish_auth 0 2024-03-25T18:27:29.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.041+0000 7ff04959a640 10 monclient: _check_auth_tickets 2024-03-25T18:27:29.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.041+0000 7ff04959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:29.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.041+0000 7ff04959a640 1 -- 172.21.15.3:0/4145898774 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7ff03c001670 con 0x7ff04c0b4ff0 2024-03-25T18:27:29.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.041+0000 7ff050d0e640 5 monclient: authenticate success, global_id 4316 2024-03-25T18:27:29.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.041+0000 7ff050d0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:29.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.041+0000 7ff04959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:29.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.041+0000 7ff04959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:29.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.041+0000 7ff04959a640 10 monclient: dump: 2024-03-25T18:27:29.044 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:29.044 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:29.044 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:29.044 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:29.044 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:29.044 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:29.044 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:29.044 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:29.044 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:29.044 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:29.045 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.041+0000 7ff050d0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:29.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.060+0000 7ff04959a640 1 -- 172.21.15.3:0/4145898774 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff03c0017d0 con 0x7ff04c0b4ff0 2024-03-25T18:27:29.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.060+0000 7ff04959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:29.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.060+0000 7ff04959a640 1 -- 172.21.15.3:0/4145898774 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7ff03c001be0 con 0x7ff04c0b4ff0 2024-03-25T18:27:29.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.060+0000 7ff04959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:29.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.060+0000 7ff04959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:29.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.060+0000 7ff04959a640 10 monclient: dump: 2024-03-25T18:27:29.062 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:29.062 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:29.062 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:29.062 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:29.062 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:29.062 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:29.062 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:29.062 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:29.062 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:29.062 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:29.063 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.060+0000 7ff050d0e640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:29.063 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.060+0000 7ff050d0e640 4 set_mon_vals no callback set 2024-03-25T18:27:29.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.061+0000 7ff050d0e640 10 monclient: shutdown 2024-03-25T18:27:29.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.061+0000 7ff050d0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:29.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.061+0000 7ff050d0e640 1 -- 172.21.15.3:0/4145898774 >> v1:172.21.15.3:6790/0 conn(0x7ff04c0b4ff0 legacy=0x7ff04c13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:29.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.061+0000 7ff050d0e640 1 -- 172.21.15.3:0/4145898774 shutdown_connections 2024-03-25T18:27:29.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.061+0000 7ff050d0e640 1 -- 172.21.15.3:0/4145898774 >> 172.21.15.3:0/4145898774 conn(0x7ff04c0617a0 msgr2=0x7ff04c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:29.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.061+0000 7ff050d0e640 1 -- 172.21.15.3:0/4145898774 shutdown_connections 2024-03-25T18:27:29.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.061+0000 7ff050d0e640 1 -- 172.21.15.3:0/4145898774 wait complete. 2024-03-25T18:27:29.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.062+0000 7ff050d0e640 10 monclient: build_initial_monmap 2024-03-25T18:27:29.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.062+0000 7ff050d0e640 10 monclient: monmap: 2024-03-25T18:27:29.065 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:27:29.065 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:29.065 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:27:29.065 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:27:29.065 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:29.065 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:27:29.065 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:27:29.065 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:27:29.065 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:27:29.065 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:29.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.063+0000 7ff050d0e640 1 librados: starting msgr at 2024-03-25T18:27:29.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.063+0000 7ff050d0e640 1 librados: starting objecter 2024-03-25T18:27:29.066 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.063+0000 7ff050d0e640 1 Processor -- start 2024-03-25T18:27:29.066 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.063+0000 7ff050d0e640 1 -- start start 2024-03-25T18:27:29.066 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.063+0000 7ff050d0e640 1 librados: setting wanted keys 2024-03-25T18:27:29.066 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.063+0000 7ff050d0e640 1 librados: calling monclient init 2024-03-25T18:27:29.066 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.063+0000 7ff050d0e640 10 monclient: init 2024-03-25T18:27:29.067 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.064+0000 7ff050d0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:29.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.064+0000 7ff050d0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:27:29.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.064+0000 7ff050d0e640 10 monclient(hunting): picked mon.noname-b con 0x7ff04c0b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:27:29.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.064+0000 7ff050d0e640 10 monclient(hunting): picked mon.noname-c con 0x7ff04c0b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:27:29.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.064+0000 7ff050d0e640 10 monclient(hunting): picked mon.noname-a con 0x7ff04c13c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:27:29.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.064+0000 7ff050d0e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff04c130c60 con 0x7ff04c13c480 2024-03-25T18:27:29.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.064+0000 7ff050d0e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff04c144800 con 0x7ff04c0b4670 2024-03-25T18:27:29.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.064+0000 7ff050d0e640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff04c144ae0 con 0x7ff04c0b4ff0 2024-03-25T18:27:29.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.064+0000 7ff050d0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:29.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.064+0000 7ff050d0e640 10 monclient(hunting): authenticate will time out at 7503.325254s 2024-03-25T18:27:29.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.066+0000 7ff04ad9d640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ff04c13c480 0x7ff04c130550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:49556/0 (socket says 172.21.15.3:49556) 2024-03-25T18:27:29.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.066+0000 7ff04ad9d640 1 -- 172.21.15.3:0/608619115 learned_addr learned my addr 172.21.15.3:0/608619115 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:27:29.070 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.069+0000 7ff03affd640 10 client.?.objecter ms_handle_connect 0x7ff04c13c480 2024-03-25T18:27:29.070 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.069+0000 7ff03affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:29.074 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.073+0000 7ff03affd640 10 client.?.objecter ms_handle_connect 0x7ff04c0b4670 2024-03-25T18:27:29.074 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.073+0000 7ff03affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:29.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.076+0000 7ff03affd640 10 client.?.objecter ms_handle_connect 0x7ff04c0b4ff0 2024-03-25T18:27:29.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.076+0000 7ff03affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:29.081 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.080+0000 7ff03affd640 1 -- 172.21.15.3:0/608619115 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719505745 0 0) 0x7ff04c130c60 con 0x7ff04c13c480 2024-03-25T18:27:29.081 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.080+0000 7ff03affd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:29.081 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.080+0000 7ff03affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:29.082 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.080+0000 7ff03affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:29.082 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.080+0000 7ff03affd640 10 monclient(hunting): my global_id is 4347 2024-03-25T18:27:29.082 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.080+0000 7ff03affd640 1 -- 172.21.15.3:0/608619115 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff02c004eb0 con 0x7ff04c13c480 2024-03-25T18:27:29.082 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.081+0000 7ff03affd640 1 -- 172.21.15.3:0/608619115 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3907195406 0 0) 0x7ff04c144800 con 0x7ff04c0b4670 2024-03-25T18:27:29.082 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.081+0000 7ff03affd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:29.083 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.081+0000 7ff03affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:29.083 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.081+0000 7ff03affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:29.083 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.081+0000 7ff03affd640 10 monclient(hunting): my global_id is 4306 2024-03-25T18:27:29.083 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.081+0000 7ff03affd640 1 -- 172.21.15.3:0/608619115 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff04c130c60 con 0x7ff04c0b4670 2024-03-25T18:27:29.098 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.097+0000 7ff03affd640 1 -- 172.21.15.3:0/608619115 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3027009725 0 0) 0x7ff02c004eb0 con 0x7ff04c13c480 2024-03-25T18:27:29.098 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.097+0000 7ff03affd640 1 -- 172.21.15.3:0/608619115 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff04c144800 con 0x7ff04c13c480 2024-03-25T18:27:29.098 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.097+0000 7ff03affd640 1 -- 172.21.15.3:0/608619115 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7ff044001c30 con 0x7ff04c13c480 2024-03-25T18:27:29.098 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.097+0000 7ff03affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:29.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.097+0000 7ff03affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:29.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.097+0000 7ff03affd640 10 monclient(hunting): dump: 2024-03-25T18:27:29.099 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:29.099 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:29.099 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:29.099 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:29.099 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:29.099 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:29.099 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:29.099 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:29.099 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:29.099 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:29.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.097+0000 7ff03affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:29.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.106+0000 7ff03affd640 1 -- 172.21.15.3:0/608619115 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2474457305 0 0) 0x7ff04c144ae0 con 0x7ff04c0b4ff0 2024-03-25T18:27:29.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.106+0000 7ff03affd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:29.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.106+0000 7ff03affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:29.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.106+0000 7ff03affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:29.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.106+0000 7ff03affd640 10 monclient(hunting): my global_id is 4319 2024-03-25T18:27:29.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.106+0000 7ff03affd640 1 -- 172.21.15.3:0/608619115 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff02c004eb0 con 0x7ff04c0b4ff0 2024-03-25T18:27:29.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.111+0000 7ff03affd640 1 -- 172.21.15.3:0/608619115 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2632456734 0 0) 0x7ff04c144800 con 0x7ff04c13c480 2024-03-25T18:27:29.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.111+0000 7ff03affd640 1 -- 172.21.15.3:0/608619115 >> v1:172.21.15.3:6790/0 conn(0x7ff04c0b4ff0 legacy=0x7ff04c12fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:29.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.111+0000 7ff03affd640 1 -- 172.21.15.3:0/608619115 >> v1:172.21.15.16:6789/0 conn(0x7ff04c0b4670 legacy=0x7ff04c145270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:29.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.111+0000 7ff03affd640 10 monclient: _finish_hunting 0 2024-03-25T18:27:29.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.111+0000 7ff03affd640 1 monclient: found mon.a 2024-03-25T18:27:29.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.111+0000 7ff03affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:29.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.111+0000 7ff03affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:27:29.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.111+0000 7ff03affd640 1 -- 172.21.15.3:0/608619115 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff04c1439a0 con 0x7ff04c13c480 2024-03-25T18:27:29.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.111+0000 7ff03affd640 10 monclient: _finish_auth 0 2024-03-25T18:27:29.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.111+0000 7ff03affd640 10 monclient: _check_auth_tickets 2024-03-25T18:27:29.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.111+0000 7ff03affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:29.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.111+0000 7ff050d0e640 5 monclient: authenticate success, global_id 4347 2024-03-25T18:27:29.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.111+0000 7ff050d0e640 10 monclient: _renew_subs 2024-03-25T18:27:29.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.111+0000 7ff050d0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:27:29.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.111+0000 7ff050d0e640 1 -- 172.21.15.3:0/608619115 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff04c1463e0 con 0x7ff04c13c480 2024-03-25T18:27:29.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.111+0000 7ff050d0e640 10 client.4347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:29.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.111+0000 7ff050d0e640 10 monclient: _renew_subs 2024-03-25T18:27:29.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.111+0000 7ff050d0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:27:29.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.111+0000 7ff050d0e640 1 -- 172.21.15.3:0/608619115 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff04c143660 con 0x7ff04c13c480 2024-03-25T18:27:29.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.112+0000 7ff050d0e640 1 librados: init done 2024-03-25T18:27:29.115 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.113+0000 7ff050d0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:29.115 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.113+0000 7ff050d0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:29.115 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.113+0000 7ff050d0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:27:29.115 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.113+0000 7ff050d0e640 1 -- 172.21.15.3:0/608619115 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff04c06dbd0 con 0x7ff04c13c480 2024-03-25T18:27:29.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.330+0000 7ff03affd640 1 -- 172.21.15.3:0/608619115 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0440020e0 con 0x7ff04c13c480 2024-03-25T18:27:29.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.330+0000 7ff03affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:29.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.330+0000 7ff048d99640 4 set_mon_vals no callback set 2024-03-25T18:27:29.433 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.432+0000 7ff03affd640 1 -- 172.21.15.3:0/608619115 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7ff044003b70 con 0x7ff04c13c480 2024-03-25T18:27:29.434 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.432+0000 7ff03affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:29.434 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.432+0000 7ff03affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:29.434 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.432+0000 7ff03affd640 10 monclient: dump: 2024-03-25T18:27:29.434 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:29.434 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:29.434 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:29.434 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:29.434 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:29.434 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:29.434 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:29.434 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:29.434 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:29.434 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:29.435 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.433+0000 7ff03affd640 1 -- 172.21.15.3:0/608619115 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7ff044003d10 con 0x7ff04c13c480 2024-03-25T18:27:29.435 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.433+0000 7ff04a59c640 10 monclient: get_auth_request con 0x7ff02c06ddb0 auth_method 0 2024-03-25T18:27:29.435 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.434+0000 7ff03affd640 10 client.4347.objecter ms_handle_connect 0x7ff02c06ddb0 2024-03-25T18:27:29.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.440+0000 7ff03affd640 1 -- 172.21.15.3:0/608619115 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7ff044054af0 con 0x7ff04c13c480 2024-03-25T18:27:29.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.440+0000 7ff03affd640 10 client.4347.objecter ms_dispatch 0x7ff04c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:29.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.440+0000 7ff03affd640 3 client.4347.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:29.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.440+0000 7ff03affd640 3 client.4347.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:29.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.440+0000 7ff03affd640 20 client.4347.objecter dump_active .. 0 homeless 2024-03-25T18:27:29.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.440+0000 7ff03affd640 1 -- 172.21.15.3:0/608619115 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff044050430 con 0x7ff04c13c480 2024-03-25T18:27:29.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.440+0000 7ff03affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:29.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.440+0000 7ff03affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:29.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.693+0000 7ff050d0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:29.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.693+0000 7ff050d0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:29.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.693+0000 7ff050d0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:27:29.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.693+0000 7ff050d0e640 1 -- 172.21.15.3:0/608619115 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff04c13d940 con 0x7ff04c13c480 2024-03-25T18:27:29.715 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.713+0000 7ff03affd640 1 -- 172.21.15.3:0/608619115 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7ff04c13d940 con 0x7ff04c13c480 2024-03-25T18:27:29.715 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.714+0000 7ff03affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:29.715 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.714+0000 7ff03affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:29.715 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:27:29.715 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:29.718 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.716+0000 7ff050d0e640 10 librados: watch_flush enter 2024-03-25T18:27:29.718 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.716+0000 7ff050d0e640 10 librados: watch_flush exit 2024-03-25T18:27:29.718 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.716+0000 7ff050d0e640 20 client.4347.objecter shutdown clearing up homeless session... 2024-03-25T18:27:29.718 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.716+0000 7ff050d0e640 10 client.4347.objecter successfully canceled tick 2024-03-25T18:27:29.718 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.716+0000 7ff050d0e640 1 -- 172.21.15.3:0/608619115 >> v1:172.21.15.3:6800/4195140966 conn(0x7ff02c06ddb0 legacy=0x7ff02c070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:29.718 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.716+0000 7ff050d0e640 10 monclient: shutdown 2024-03-25T18:27:29.718 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.716+0000 7ff050d0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:29.718 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.716+0000 7ff050d0e640 1 -- 172.21.15.3:0/608619115 >> v1:172.21.15.3:6789/0 conn(0x7ff04c13c480 legacy=0x7ff04c130550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:29.718 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.716+0000 7ff050d0e640 1 -- 172.21.15.3:0/608619115 shutdown_connections 2024-03-25T18:27:29.718 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.716+0000 7ff050d0e640 1 -- 172.21.15.3:0/608619115 >> 172.21.15.3:0/608619115 conn(0x7ff04c0617a0 msgr2=0x7ff04c140670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:29.718 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.716+0000 7ff050d0e640 1 -- 172.21.15.3:0/608619115 shutdown_connections 2024-03-25T18:27:29.718 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.716+0000 7ff050d0e640 1 -- 172.21.15.3:0/608619115 wait complete. 2024-03-25T18:27:29.719 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:29.716+0000 7ff050d0e640 1 librados: shutdown 2024-03-25T18:27:29.728 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:36.535 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:36.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.650+0000 7f622a96d640 10 monclient: get_monmap_and_config 2024-03-25T18:27:36.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.650+0000 7f622a96d640 10 monclient: build_initial_monmap 2024-03-25T18:27:36.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.650+0000 7f622a96d640 10 monclient: monmap: 2024-03-25T18:27:36.652 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:27:36.652 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:36.652 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:27:36.651706+0000 2024-03-25T18:27:36.652 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:27:36.651706+0000 2024-03-25T18:27:36.652 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:36.652 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:36.652 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:27:36.652 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:27:36.652 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:27:36.652 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:36.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.651+0000 7f622a96d640 1 Processor -- start 2024-03-25T18:27:36.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.651+0000 7f622a96d640 1 -- start start 2024-03-25T18:27:36.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.651+0000 7f622a96d640 10 monclient: init 2024-03-25T18:27:36.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.651+0000 7f622a96d640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:36.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.651+0000 7f622a96d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:27:36.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.651+0000 7f622a96d640 10 monclient(hunting): picked mon.noname-b con 0x7f62240b4e70 addr v1:172.21.15.3:6790/0 2024-03-25T18:27:36.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.651+0000 7f622a96d640 10 monclient(hunting): picked mon.noname-a con 0x7f62240b44b0 addr v1:172.21.15.3:6789/0 2024-03-25T18:27:36.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.652+0000 7f622a96d640 10 monclient(hunting): picked mon.noname-c con 0x7f62240b1b00 addr v1:172.21.15.16:6789/0 2024-03-25T18:27:36.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.652+0000 7f622a96d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62240a96f0 con 0x7f62240b44b0 2024-03-25T18:27:36.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.652+0000 7f622a96d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62240ab0e0 con 0x7f62240b1b00 2024-03-25T18:27:36.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.652+0000 7f622a96d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6224145f70 con 0x7f62240b4e70 2024-03-25T18:27:36.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.652+0000 7f622a96d640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:36.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.652+0000 7f622a96d640 10 monclient(hunting): authenticate will time out at 7510.912393s 2024-03-25T18:27:36.656 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.655+0000 7f6223fff640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f62240b4e70 0x7f622413f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:40592/0 (socket says 172.21.15.3:40592) 2024-03-25T18:27:36.656 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.655+0000 7f6223fff640 1 -- 172.21.15.3:0/4098313329 learned_addr learned my addr 172.21.15.3:0/4098313329 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:27:36.663 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.662+0000 7f6222ffd640 1 -- 172.21.15.3:0/4098313329 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3160353728 0 0) 0x7f6224145f70 con 0x7f62240b4e70 2024-03-25T18:27:36.663 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.662+0000 7f6222ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:36.663 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.662+0000 7f6222ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:36.663 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.662+0000 7f6222ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:36.663 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.662+0000 7f6222ffd640 10 monclient(hunting): my global_id is 4322 2024-03-25T18:27:36.663 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.662+0000 7f6222ffd640 1 -- 172.21.15.3:0/4098313329 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f620c003520 con 0x7f62240b4e70 2024-03-25T18:27:36.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.672+0000 7f6222ffd640 1 -- 172.21.15.3:0/4098313329 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1774910420 0 0) 0x7f62240a96f0 con 0x7f62240b44b0 2024-03-25T18:27:36.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.672+0000 7f6222ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:36.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.672+0000 7f6222ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:36.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.672+0000 7f6222ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:36.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.672+0000 7f6222ffd640 10 monclient(hunting): my global_id is 4350 2024-03-25T18:27:36.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.672+0000 7f6222ffd640 1 -- 172.21.15.3:0/4098313329 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6224145f70 con 0x7f62240b44b0 2024-03-25T18:27:36.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.675+0000 7f6222ffd640 1 -- 172.21.15.3:0/4098313329 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442680037 0 0) 0x7f62240ab0e0 con 0x7f62240b1b00 2024-03-25T18:27:36.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.675+0000 7f6222ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:36.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.675+0000 7f6222ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:36.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.675+0000 7f6222ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:36.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.675+0000 7f6222ffd640 10 monclient(hunting): my global_id is 4309 2024-03-25T18:27:36.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.675+0000 7f6222ffd640 1 -- 172.21.15.3:0/4098313329 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62240a96f0 con 0x7f62240b1b00 2024-03-25T18:27:36.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.675+0000 7f6222ffd640 1 -- 172.21.15.3:0/4098313329 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3843873801 0 0) 0x7f620c003520 con 0x7f62240b4e70 2024-03-25T18:27:36.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.675+0000 7f6222ffd640 1 -- 172.21.15.3:0/4098313329 >> v1:172.21.15.16:6789/0 conn(0x7f62240b1b00 legacy=0x7f622414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:36.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.675+0000 7f6222ffd640 1 -- 172.21.15.3:0/4098313329 >> v1:172.21.15.3:6789/0 conn(0x7f62240b44b0 legacy=0x7f6224145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:36.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.675+0000 7f6222ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:27:36.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.675+0000 7f6222ffd640 1 monclient: found mon.noname-b 2024-03-25T18:27:36.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.675+0000 7f6222ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:36.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.675+0000 7f6222ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-25T18:27:36.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.675+0000 7f6222ffd640 1 -- 172.21.15.3:0/4098313329 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62240a9b80 con 0x7f62240b4e70 2024-03-25T18:27:36.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.675+0000 7f6222ffd640 10 monclient: _finish_auth 0 2024-03-25T18:27:36.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.675+0000 7f6222ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:27:36.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.675+0000 7f6222ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:36.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.675+0000 7f6222ffd640 1 -- 172.21.15.3:0/4098313329 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f6214001690 con 0x7f62240b4e70 2024-03-25T18:27:36.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.675+0000 7f6222ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:36.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.675+0000 7f6222ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:36.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.675+0000 7f6222ffd640 10 monclient: dump: 2024-03-25T18:27:36.678 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:36.678 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:36.678 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:36.679 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:36.679 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:36.679 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:36.679 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:36.679 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:36.679 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:36.679 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:36.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.675+0000 7f622a96d640 5 monclient: authenticate success, global_id 4322 2024-03-25T18:27:36.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.675+0000 7f622a96d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:36.699 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.698+0000 7f6222ffd640 1 -- 172.21.15.3:0/4098313329 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62140017f0 con 0x7f62240b4e70 2024-03-25T18:27:36.699 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.698+0000 7f6222ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:36.699 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.698+0000 7f6222ffd640 1 -- 172.21.15.3:0/4098313329 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f6214001c00 con 0x7f62240b4e70 2024-03-25T18:27:36.699 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.698+0000 7f6222ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:36.699 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.698+0000 7f6222ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:36.699 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.698+0000 7f6222ffd640 10 monclient: dump: 2024-03-25T18:27:36.699 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:36.699 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:36.699 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:36.699 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:36.700 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:36.700 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:36.700 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:36.700 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:36.700 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:36.700 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:36.700 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.698+0000 7f622a96d640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:36.700 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.698+0000 7f622a96d640 4 set_mon_vals no callback set 2024-03-25T18:27:36.702 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.698+0000 7f622a96d640 10 monclient: shutdown 2024-03-25T18:27:36.702 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.698+0000 7f622a96d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:36.702 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.698+0000 7f622a96d640 1 -- 172.21.15.3:0/4098313329 >> v1:172.21.15.3:6790/0 conn(0x7f62240b4e70 legacy=0x7f622413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:36.702 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.699+0000 7f622a96d640 1 -- 172.21.15.3:0/4098313329 shutdown_connections 2024-03-25T18:27:36.702 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.699+0000 7f622a96d640 1 -- 172.21.15.3:0/4098313329 >> 172.21.15.3:0/4098313329 conn(0x7f62240617a0 msgr2=0x7f6224061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:36.702 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.699+0000 7f622a96d640 1 -- 172.21.15.3:0/4098313329 shutdown_connections 2024-03-25T18:27:36.702 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.699+0000 7f622a96d640 1 -- 172.21.15.3:0/4098313329 wait complete. 2024-03-25T18:27:36.703 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.699+0000 7f622a96d640 10 monclient: build_initial_monmap 2024-03-25T18:27:36.703 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.699+0000 7f622a96d640 10 monclient: monmap: 2024-03-25T18:27:36.703 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:27:36.703 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:36.703 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:27:36.703 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:27:36.703 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:36.703 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:27:36.703 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:27:36.703 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:27:36.703 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:27:36.703 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:36.703 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.700+0000 7f622a96d640 1 librados: starting msgr at 2024-03-25T18:27:36.703 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.700+0000 7f622a96d640 1 librados: starting objecter 2024-03-25T18:27:36.704 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.700+0000 7f622a96d640 1 Processor -- start 2024-03-25T18:27:36.704 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.700+0000 7f622a96d640 1 -- start start 2024-03-25T18:27:36.704 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.700+0000 7f622a96d640 1 librados: setting wanted keys 2024-03-25T18:27:36.704 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.700+0000 7f622a96d640 1 librados: calling monclient init 2024-03-25T18:27:36.704 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.700+0000 7f622a96d640 10 monclient: init 2024-03-25T18:27:36.704 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.700+0000 7f622a96d640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:36.704 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.700+0000 7f622a96d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:27:36.704 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.700+0000 7f622a96d640 10 monclient(hunting): picked mon.noname-b con 0x7f62240b1b00 addr v1:172.21.15.16:6789/0 2024-03-25T18:27:36.704 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.700+0000 7f622a96d640 10 monclient(hunting): picked mon.noname-c con 0x7f62240b44b0 addr v1:172.21.15.3:6790/0 2024-03-25T18:27:36.704 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.701+0000 7f622a96d640 10 monclient(hunting): picked mon.noname-a con 0x7f62240b4e70 addr v1:172.21.15.3:6789/0 2024-03-25T18:27:36.704 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.701+0000 7f622a96d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62241369d0 con 0x7f62240b4e70 2024-03-25T18:27:36.704 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.701+0000 7f622a96d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6224136cb0 con 0x7f62240b1b00 2024-03-25T18:27:36.704 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.701+0000 7f622a96d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6224136f90 con 0x7f62240b44b0 2024-03-25T18:27:36.704 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.701+0000 7f622a96d640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:36.705 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.701+0000 7f622a96d640 10 monclient(hunting): authenticate will time out at 7510.961455s 2024-03-25T18:27:36.711 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.710+0000 7f6223fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f62240b1b00 0x7f62241350e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:51310/0 (socket says 172.21.15.3:51310) 2024-03-25T18:27:36.711 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.710+0000 7f6223fff640 1 -- 172.21.15.3:0/2369932190 learned_addr learned my addr 172.21.15.3:0/2369932190 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:27:36.713 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.712+0000 7f6220ff9640 10 client.?.objecter ms_handle_connect 0x7f62240b44b0 2024-03-25T18:27:36.714 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.712+0000 7f6220ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:36.714 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.713+0000 7f6220ff9640 10 client.?.objecter ms_handle_connect 0x7f62240b1b00 2024-03-25T18:27:36.714 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.713+0000 7f6220ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:36.720 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.719+0000 7f6220ff9640 10 client.?.objecter ms_handle_connect 0x7f62240b4e70 2024-03-25T18:27:36.720 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.719+0000 7f6220ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:36.724 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.723+0000 7f6220ff9640 1 -- 172.21.15.3:0/2369932190 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2203065298 0 0) 0x7f6224136cb0 con 0x7f62240b1b00 2024-03-25T18:27:36.724 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.723+0000 7f6220ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:36.724 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.723+0000 7f6220ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:36.724 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.723+0000 7f6220ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:36.724 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.723+0000 7f6220ff9640 10 monclient(hunting): my global_id is 4312 2024-03-25T18:27:36.725 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.723+0000 7f6220ff9640 1 -- 172.21.15.3:0/2369932190 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6210004eb0 con 0x7f62240b1b00 2024-03-25T18:27:36.731 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.730+0000 7f6220ff9640 1 -- 172.21.15.3:0/2369932190 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1810401366 0 0) 0x7f62241369d0 con 0x7f62240b4e70 2024-03-25T18:27:36.731 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.730+0000 7f6220ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:36.731 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.730+0000 7f6220ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:36.731 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.730+0000 7f6220ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:36.731 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.730+0000 7f6220ff9640 10 monclient(hunting): my global_id is 4353 2024-03-25T18:27:36.731 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.730+0000 7f6220ff9640 1 -- 172.21.15.3:0/2369932190 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6224136cb0 con 0x7f62240b4e70 2024-03-25T18:27:36.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.744+0000 7f6220ff9640 1 -- 172.21.15.3:0/2369932190 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3476936929 0 0) 0x7f6224136cb0 con 0x7f62240b4e70 2024-03-25T18:27:36.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.744+0000 7f6220ff9640 1 -- 172.21.15.3:0/2369932190 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62241369d0 con 0x7f62240b4e70 2024-03-25T18:27:36.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.744+0000 7f6220ff9640 1 -- 172.21.15.3:0/2369932190 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f621c001bf0 con 0x7f62240b4e70 2024-03-25T18:27:36.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.744+0000 7f6220ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:36.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.744+0000 7f6220ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:36.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.744+0000 7f6220ff9640 10 monclient(hunting): dump: 2024-03-25T18:27:36.746 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:36.746 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:36.746 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:36.746 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:36.746 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:36.746 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:36.746 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:36.746 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:36.747 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:36.747 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:36.747 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.744+0000 7f6220ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:36.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.755+0000 7f6220ff9640 1 -- 172.21.15.3:0/2369932190 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4216451722 0 0) 0x7f62241369d0 con 0x7f62240b4e70 2024-03-25T18:27:36.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.755+0000 7f6220ff9640 1 -- 172.21.15.3:0/2369932190 >> v1:172.21.15.3:6790/0 conn(0x7f62240b44b0 legacy=0x7f6224137770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:36.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.755+0000 7f6220ff9640 1 -- 172.21.15.3:0/2369932190 >> v1:172.21.15.16:6789/0 conn(0x7f62240b1b00 legacy=0x7f62241350e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:36.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.756+0000 7f6220ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:27:36.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.756+0000 7f6220ff9640 1 monclient: found mon.a 2024-03-25T18:27:36.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.756+0000 7f6220ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:36.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.756+0000 7f6220ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:27:36.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.756+0000 7f6220ff9640 1 -- 172.21.15.3:0/2369932190 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6224137e80 con 0x7f62240b4e70 2024-03-25T18:27:36.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.756+0000 7f6220ff9640 10 monclient: _finish_auth 0 2024-03-25T18:27:36.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.756+0000 7f6220ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:27:36.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.756+0000 7f6220ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:36.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.756+0000 7f622a96d640 5 monclient: authenticate success, global_id 4353 2024-03-25T18:27:36.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.756+0000 7f622a96d640 10 monclient: _renew_subs 2024-03-25T18:27:36.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.756+0000 7f622a96d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:27:36.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.756+0000 7f622a96d640 1 -- 172.21.15.3:0/2369932190 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62241447c0 con 0x7f62240b4e70 2024-03-25T18:27:36.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.756+0000 7f622a96d640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:36.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.756+0000 7f622a96d640 10 monclient: _renew_subs 2024-03-25T18:27:36.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.756+0000 7f622a96d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:27:36.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.756+0000 7f622a96d640 1 -- 172.21.15.3:0/2369932190 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6224143320 con 0x7f62240b4e70 2024-03-25T18:27:36.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.756+0000 7f622a96d640 1 librados: init done 2024-03-25T18:27:36.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.756+0000 7f622a96d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:36.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.757+0000 7f622a96d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:36.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.757+0000 7f622a96d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:27:36.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:36.757+0000 7f622a96d640 1 -- 172.21.15.3:0/2369932190 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f622404f730 con 0x7f62240b4e70 2024-03-25T18:27:37.027 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.026+0000 7f6220ff9640 1 -- 172.21.15.3:0/2369932190 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f621c002080 con 0x7f62240b4e70 2024-03-25T18:27:37.027 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.026+0000 7f6220ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:37.027 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.026+0000 7f6220ff9640 1 -- 172.21.15.3:0/2369932190 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f621c003bf0 con 0x7f62240b4e70 2024-03-25T18:27:37.027 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.026+0000 7f6220ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:37.027 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.026+0000 7f6220ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:37.027 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.026+0000 7f6220ff9640 10 monclient: dump: 2024-03-25T18:27:37.028 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:37.028 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:37.028 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:37.028 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:37.028 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:37.028 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:37.028 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:37.028 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:37.028 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:37.028 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:37.028 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.026+0000 7f62227fc640 4 set_mon_vals no callback set 2024-03-25T18:27:37.028 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.026+0000 7f6220ff9640 1 -- 172.21.15.3:0/2369932190 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f621c003e10 con 0x7f62240b4e70 2024-03-25T18:27:37.029 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.027+0000 7f6223fff640 10 monclient: get_auth_request con 0x7f621006dd00 auth_method 0 2024-03-25T18:27:37.029 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.027+0000 7f6220ff9640 10 client.4353.objecter ms_handle_connect 0x7f621006dd00 2024-03-25T18:27:37.034 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.032+0000 7f6220ff9640 1 -- 172.21.15.3:0/2369932190 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f621c053890 con 0x7f62240b4e70 2024-03-25T18:27:37.034 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.032+0000 7f6220ff9640 10 client.4353.objecter ms_dispatch 0x7f6224000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:37.034 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.032+0000 7f6220ff9640 3 client.4353.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:37.034 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.032+0000 7f6220ff9640 3 client.4353.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:37.034 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.032+0000 7f6220ff9640 20 client.4353.objecter dump_active .. 0 homeless 2024-03-25T18:27:37.034 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.032+0000 7f6220ff9640 1 -- 172.21.15.3:0/2369932190 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f621c04f1d0 con 0x7f62240b4e70 2024-03-25T18:27:37.034 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.032+0000 7f6220ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:37.034 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.032+0000 7f6220ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:37.274 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.273+0000 7f622a96d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:37.274 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.273+0000 7f622a96d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:37.275 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.273+0000 7f622a96d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:27:37.275 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.273+0000 7f622a96d640 1 -- 172.21.15.3:0/2369932190 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6224136080 con 0x7f62240b4e70 2024-03-25T18:27:37.311 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.309+0000 7f6220ff9640 1 -- 172.21.15.3:0/2369932190 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f621c0512d0 con 0x7f62240b4e70 2024-03-25T18:27:37.311 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.310+0000 7f6220ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:37.311 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.310+0000 7f6220ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:37.311 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:27:37.311 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:37.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.312+0000 7f622a96d640 10 librados: watch_flush enter 2024-03-25T18:27:37.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.312+0000 7f622a96d640 10 librados: watch_flush exit 2024-03-25T18:27:37.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.312+0000 7f622a96d640 20 client.4353.objecter shutdown clearing up homeless session... 2024-03-25T18:27:37.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.312+0000 7f622a96d640 10 client.4353.objecter successfully canceled tick 2024-03-25T18:27:37.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.312+0000 7f622a96d640 1 -- 172.21.15.3:0/2369932190 >> v1:172.21.15.3:6800/4195140966 conn(0x7f621006dd00 legacy=0x7f62100701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:37.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.312+0000 7f622a96d640 10 monclient: shutdown 2024-03-25T18:27:37.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.312+0000 7f622a96d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:37.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.312+0000 7f622a96d640 1 -- 172.21.15.3:0/2369932190 >> v1:172.21.15.3:6789/0 conn(0x7f62240b4e70 legacy=0x7f62241e2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:37.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.312+0000 7f622a96d640 1 -- 172.21.15.3:0/2369932190 shutdown_connections 2024-03-25T18:27:37.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.312+0000 7f622a96d640 1 -- 172.21.15.3:0/2369932190 >> 172.21.15.3:0/2369932190 conn(0x7f62240617a0 msgr2=0x7f6224133ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:37.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.312+0000 7f622a96d640 1 -- 172.21.15.3:0/2369932190 shutdown_connections 2024-03-25T18:27:37.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.312+0000 7f622a96d640 1 -- 172.21.15.3:0/2369932190 wait complete. 2024-03-25T18:27:37.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:37.312+0000 7f622a96d640 1 librados: shutdown 2024-03-25T18:27:37.325 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:41.139 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:02:54.868 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:41.204 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:02:54.932 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:41.244 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:02:54.973 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:41.256 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:02:54.985 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:41.261 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:02:54.990 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:41.275 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:02:54.967 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:41.286 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:02:54.978 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:41.299 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:02:55.028 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:41.321 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:02:55.013 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:44.132 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:44.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.246+0000 7f24255e8640 10 monclient: get_monmap_and_config 2024-03-25T18:27:44.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.246+0000 7f24255e8640 10 monclient: build_initial_monmap 2024-03-25T18:27:44.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.246+0000 7f24255e8640 10 monclient: monmap: 2024-03-25T18:27:44.247 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:27:44.247 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:44.247 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:27:44.247251+0000 2024-03-25T18:27:44.247 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:27:44.247251+0000 2024-03-25T18:27:44.247 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:44.247 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:44.248 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:27:44.248 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:27:44.248 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:27:44.248 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:44.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.246+0000 7f24255e8640 1 Processor -- start 2024-03-25T18:27:44.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.247+0000 7f24255e8640 1 -- start start 2024-03-25T18:27:44.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.247+0000 7f24255e8640 10 monclient: init 2024-03-25T18:27:44.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.247+0000 7f24255e8640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:44.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.247+0000 7f24255e8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:27:44.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.247+0000 7f24255e8640 10 monclient(hunting): picked mon.noname-b con 0x7f2420061830 addr v1:172.21.15.3:6790/0 2024-03-25T18:27:44.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.247+0000 7f24255e8640 10 monclient(hunting): picked mon.noname-c con 0x7f242006fd50 addr v1:172.21.15.16:6789/0 2024-03-25T18:27:44.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.247+0000 7f24255e8640 10 monclient(hunting): picked mon.noname-a con 0x7f242013c740 addr v1:172.21.15.3:6789/0 2024-03-25T18:27:44.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.247+0000 7f24255e8640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f242006f8c0 con 0x7f242013c740 2024-03-25T18:27:44.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.247+0000 7f24255e8640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2420072010 con 0x7f242006fd50 2024-03-25T18:27:44.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.247+0000 7f24255e8640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2420072b40 con 0x7f2420061830 2024-03-25T18:27:44.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.247+0000 7f24255e8640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:44.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.247+0000 7f24255e8640 10 monclient(hunting): authenticate will time out at 7518.508268s 2024-03-25T18:27:44.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.250+0000 7f241e7fc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f242006fd50 0x7f242013c030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:56490/0 (socket says 172.21.15.3:56490) 2024-03-25T18:27:44.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.250+0000 7f241e7fc640 1 -- 172.21.15.3:0/1795603828 learned_addr learned my addr 172.21.15.3:0/1795603828 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:27:44.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.253+0000 7f241dffb640 1 -- 172.21.15.3:0/1795603828 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1310272946 0 0) 0x7f2420072010 con 0x7f242006fd50 2024-03-25T18:27:44.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.253+0000 7f241dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:27:44.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.253+0000 7f241dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:44.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.253+0000 7f241dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:44.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.253+0000 7f241dffb640 10 monclient(hunting): my global_id is 4315 2024-03-25T18:27:44.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.253+0000 7f241dffb640 1 -- 172.21.15.3:0/1795603828 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2408005730 con 0x7f242006fd50 2024-03-25T18:27:44.261 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.260+0000 7f241dffb640 1 -- 172.21.15.3:0/1795603828 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3362735766 0 0) 0x7f2408005730 con 0x7f242006fd50 2024-03-25T18:27:44.262 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.260+0000 7f241dffb640 1 -- 172.21.15.3:0/1795603828 >> v1:172.21.15.3:6790/0 conn(0x7f2420061830 legacy=0x7f2420072430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:27:44.262 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.260+0000 7f241dffb640 1 -- 172.21.15.3:0/1795603828 >> v1:172.21.15.3:6789/0 conn(0x7f242013c740 legacy=0x7f242014a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:27:44.262 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.260+0000 7f241dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:27:44.262 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.260+0000 7f241dffb640 1 monclient: found mon.noname-c 2024-03-25T18:27:44.262 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.260+0000 7f241dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:44.262 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.260+0000 7f241dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:27:44.262 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.260+0000 7f241dffb640 1 -- 172.21.15.3:0/1795603828 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2420072cd0 con 0x7f242006fd50 2024-03-25T18:27:44.262 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.260+0000 7f241dffb640 10 monclient: _finish_auth 0 2024-03-25T18:27:44.262 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.260+0000 7f241dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:27:44.262 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.260+0000 7f241dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:44.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.260+0000 7f241dffb640 1 -- 172.21.15.3:0/1795603828 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f2414001690 con 0x7f242006fd50 2024-03-25T18:27:44.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.260+0000 7f241dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:44.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.260+0000 7f241dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:44.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.260+0000 7f241dffb640 10 monclient: dump: 2024-03-25T18:27:44.263 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:44.263 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:44.263 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:44.263 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:44.263 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:44.263 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:44.263 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:44.263 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:44.263 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:44.263 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:44.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.260+0000 7f24255e8640 5 monclient: authenticate success, global_id 4315 2024-03-25T18:27:44.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.260+0000 7f24255e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:44.271 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.269+0000 7f241dffb640 1 -- 172.21.15.3:0/1795603828 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24140017f0 con 0x7f242006fd50 2024-03-25T18:27:44.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.269+0000 7f241dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:44.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.269+0000 7f241dffb640 1 -- 172.21.15.3:0/1795603828 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f2414001c40 con 0x7f242006fd50 2024-03-25T18:27:44.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.269+0000 7f241dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:44.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.269+0000 7f241dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:44.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.269+0000 7f241dffb640 10 monclient: dump: 2024-03-25T18:27:44.272 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:44.272 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:44.272 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:44.273 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:44.273 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:44.273 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:44.273 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:44.273 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:44.274 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:44.274 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:44.274 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.269+0000 7f24255e8640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:44.274 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.269+0000 7f24255e8640 4 set_mon_vals no callback set 2024-03-25T18:27:44.274 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.270+0000 7f24255e8640 10 monclient: shutdown 2024-03-25T18:27:44.274 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.270+0000 7f24255e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:44.274 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.270+0000 7f24255e8640 1 -- 172.21.15.3:0/1795603828 >> v1:172.21.15.16:6789/0 conn(0x7f242006fd50 legacy=0x7f242013c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:44.274 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.270+0000 7f24255e8640 1 -- 172.21.15.3:0/1795603828 shutdown_connections 2024-03-25T18:27:44.274 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.270+0000 7f24255e8640 1 -- 172.21.15.3:0/1795603828 >> 172.21.15.3:0/1795603828 conn(0x7f24200b16c0 msgr2=0x7f242006aba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:44.275 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.270+0000 7f24255e8640 1 -- 172.21.15.3:0/1795603828 shutdown_connections 2024-03-25T18:27:44.275 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.270+0000 7f24255e8640 1 -- 172.21.15.3:0/1795603828 wait complete. 2024-03-25T18:27:44.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.271+0000 7f24255e8640 10 monclient: build_initial_monmap 2024-03-25T18:27:44.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.271+0000 7f24255e8640 10 monclient: monmap: 2024-03-25T18:27:44.276 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:27:44.276 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:44.277 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:27:44.277 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:27:44.277 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:44.277 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:27:44.277 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:27:44.277 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:27:44.277 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:27:44.277 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:44.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.271+0000 7f24255e8640 1 librados: starting msgr at 2024-03-25T18:27:44.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.271+0000 7f24255e8640 1 librados: starting objecter 2024-03-25T18:27:44.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.271+0000 7f24255e8640 1 Processor -- start 2024-03-25T18:27:44.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.271+0000 7f24255e8640 1 -- start start 2024-03-25T18:27:44.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.271+0000 7f24255e8640 1 librados: setting wanted keys 2024-03-25T18:27:44.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.271+0000 7f24255e8640 1 librados: calling monclient init 2024-03-25T18:27:44.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.271+0000 7f24255e8640 10 monclient: init 2024-03-25T18:27:44.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.272+0000 7f24255e8640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:44.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.272+0000 7f24255e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:27:44.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.272+0000 7f24255e8640 10 monclient(hunting): picked mon.noname-b con 0x7f2420061830 addr v1:172.21.15.16:6789/0 2024-03-25T18:27:44.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.272+0000 7f24255e8640 10 monclient(hunting): picked mon.noname-a con 0x7f242006fd50 addr v1:172.21.15.3:6789/0 2024-03-25T18:27:44.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.272+0000 7f24255e8640 10 monclient(hunting): picked mon.noname-c con 0x7f242013c740 addr v1:172.21.15.3:6790/0 2024-03-25T18:27:44.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.272+0000 7f24255e8640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f242013d2f0 con 0x7f242006fd50 2024-03-25T18:27:44.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.272+0000 7f24255e8640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f242013d810 con 0x7f2420061830 2024-03-25T18:27:44.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.272+0000 7f24255e8640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f242013ada0 con 0x7f242013c740 2024-03-25T18:27:44.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.272+0000 7f24255e8640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:44.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.272+0000 7f24255e8640 10 monclient(hunting): authenticate will time out at 7518.533177s 2024-03-25T18:27:44.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.279+0000 7f241effd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2420061830 0x7f2420133730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:56516/0 (socket says 172.21.15.3:56516) 2024-03-25T18:27:44.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.279+0000 7f241effd640 1 -- 172.21.15.3:0/3701744401 learned_addr learned my addr 172.21.15.3:0/3701744401 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:27:44.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.285+0000 7f24077fe640 10 client.?.objecter ms_handle_connect 0x7f2420061830 2024-03-25T18:27:44.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.285+0000 7f24077fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:44.292 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.291+0000 7f24077fe640 10 client.?.objecter ms_handle_connect 0x7f242006fd50 2024-03-25T18:27:44.292 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.291+0000 7f24077fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:44.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.292+0000 7f24077fe640 10 client.?.objecter ms_handle_connect 0x7f242013c740 2024-03-25T18:27:44.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.292+0000 7f24077fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:44.298 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.297+0000 7f24077fe640 1 -- 172.21.15.3:0/3701744401 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572882444 0 0) 0x7f242013d810 con 0x7f2420061830 2024-03-25T18:27:44.298 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.297+0000 7f24077fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:44.298 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.297+0000 7f24077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:44.298 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.297+0000 7f24077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:44.298 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.297+0000 7f24077fe640 10 monclient(hunting): my global_id is 4318 2024-03-25T18:27:44.298 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.297+0000 7f24077fe640 1 -- 172.21.15.3:0/3701744401 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f240c004eb0 con 0x7f2420061830 2024-03-25T18:27:44.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.302+0000 7f24077fe640 1 -- 172.21.15.3:0/3701744401 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3345648153 0 0) 0x7f242013ada0 con 0x7f242013c740 2024-03-25T18:27:44.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.302+0000 7f24077fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:44.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.302+0000 7f24077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:44.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.302+0000 7f24077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:44.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.302+0000 7f24077fe640 10 monclient(hunting): my global_id is 4325 2024-03-25T18:27:44.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.302+0000 7f24077fe640 1 -- 172.21.15.3:0/3701744401 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f242013d810 con 0x7f242013c740 2024-03-25T18:27:44.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.307+0000 7f24077fe640 1 -- 172.21.15.3:0/3701744401 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2743769815 0 0) 0x7f242013d2f0 con 0x7f242006fd50 2024-03-25T18:27:44.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.307+0000 7f24077fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:44.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.307+0000 7f24077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:44.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.307+0000 7f24077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:44.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.307+0000 7f24077fe640 10 monclient(hunting): my global_id is 4356 2024-03-25T18:27:44.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.307+0000 7f24077fe640 1 -- 172.21.15.3:0/3701744401 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f242013ada0 con 0x7f242006fd50 2024-03-25T18:27:44.309 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.308+0000 7f24077fe640 1 -- 172.21.15.3:0/3701744401 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 8171687 0 0) 0x7f240c004eb0 con 0x7f2420061830 2024-03-25T18:27:44.309 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.308+0000 7f24077fe640 1 -- 172.21.15.3:0/3701744401 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f242013d2f0 con 0x7f2420061830 2024-03-25T18:27:44.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.308+0000 7f24077fe640 1 -- 172.21.15.3:0/3701744401 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f2410001960 con 0x7f2420061830 2024-03-25T18:27:44.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.308+0000 7f24077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:44.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.308+0000 7f24077fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:44.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.308+0000 7f24077fe640 10 monclient(hunting): dump: 2024-03-25T18:27:44.310 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:44.310 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:44.310 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:44.310 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:44.310 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:44.310 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:44.310 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:44.310 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:44.310 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:44.310 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:44.311 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.308+0000 7f24077fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:44.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.313+0000 7f24077fe640 1 -- 172.21.15.3:0/3701744401 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2338358123 0 0) 0x7f242013d810 con 0x7f242013c740 2024-03-25T18:27:44.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.313+0000 7f24077fe640 1 -- 172.21.15.3:0/3701744401 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f240c004eb0 con 0x7f242013c740 2024-03-25T18:27:44.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.313+0000 7f24077fe640 1 -- 172.21.15.3:0/3701744401 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f2418001a70 con 0x7f242013c740 2024-03-25T18:27:44.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.313+0000 7f24077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:44.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.313+0000 7f24077fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:44.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.313+0000 7f24077fe640 10 monclient(hunting): dump: 2024-03-25T18:27:44.314 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:44.314 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:44.314 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:44.314 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:44.314 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:44.315 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:44.315 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:44.315 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:44.315 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:44.315 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:44.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.315+0000 7f24077fe640 1 -- 172.21.15.3:0/3701744401 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1817257175 0 0) 0x7f242013d2f0 con 0x7f2420061830 2024-03-25T18:27:44.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.315+0000 7f24077fe640 1 -- 172.21.15.3:0/3701744401 >> v1:172.21.15.3:6790/0 conn(0x7f242013c740 legacy=0x7f24201e3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:44.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.315+0000 7f24077fe640 1 -- 172.21.15.3:0/3701744401 >> v1:172.21.15.3:6789/0 conn(0x7f242006fd50 legacy=0x7f24201d7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:44.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.315+0000 7f24077fe640 10 monclient: _finish_hunting 0 2024-03-25T18:27:44.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.315+0000 7f24077fe640 1 monclient: found mon.b 2024-03-25T18:27:44.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.315+0000 7f24077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:44.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.315+0000 7f24077fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:27:44.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.315+0000 7f24077fe640 1 -- 172.21.15.3:0/3701744401 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f242013af30 con 0x7f2420061830 2024-03-25T18:27:44.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.315+0000 7f24077fe640 10 monclient: _finish_auth 0 2024-03-25T18:27:44.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.315+0000 7f24077fe640 10 monclient: _check_auth_tickets 2024-03-25T18:27:44.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.315+0000 7f24077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:44.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.315+0000 7f24255e8640 5 monclient: authenticate success, global_id 4318 2024-03-25T18:27:44.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.315+0000 7f24255e8640 10 monclient: _renew_subs 2024-03-25T18:27:44.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.315+0000 7f24255e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:27:44.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.315+0000 7f24255e8640 1 -- 172.21.15.3:0/3701744401 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24200b0c70 con 0x7f2420061830 2024-03-25T18:27:44.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.315+0000 7f24255e8640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:44.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.315+0000 7f24255e8640 10 monclient: _renew_subs 2024-03-25T18:27:44.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.315+0000 7f24255e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:27:44.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.315+0000 7f24255e8640 1 -- 172.21.15.3:0/3701744401 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f242013d040 con 0x7f2420061830 2024-03-25T18:27:44.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.315+0000 7f24255e8640 1 librados: init done 2024-03-25T18:27:44.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.316+0000 7f24255e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:44.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.316+0000 7f24255e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:44.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.316+0000 7f24255e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:27:44.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.316+0000 7f24255e8640 1 -- 172.21.15.3:0/3701744401 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24200aecf0 con 0x7f2420061830 2024-03-25T18:27:44.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.598+0000 7f24077fe640 1 -- 172.21.15.3:0/3701744401 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24100035f0 con 0x7f2420061830 2024-03-25T18:27:44.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.598+0000 7f24077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:44.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.598+0000 7f24077fe640 1 -- 172.21.15.3:0/3701744401 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f24100037a0 con 0x7f2420061830 2024-03-25T18:27:44.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.598+0000 7f24077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:44.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.598+0000 7f24077fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:44.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.598+0000 7f24077fe640 10 monclient: dump: 2024-03-25T18:27:44.600 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:44.600 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:44.600 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:44.600 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:44.600 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:44.600 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:44.600 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:44.600 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:44.601 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:44.601 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:44.601 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.598+0000 7f241d7fa640 4 set_mon_vals no callback set 2024-03-25T18:27:44.601 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.599+0000 7f24077fe640 1 -- 172.21.15.3:0/3701744401 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f24100039c0 con 0x7f2420061830 2024-03-25T18:27:44.601 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.600+0000 7f241e7fc640 10 monclient: get_auth_request con 0x7f240c06df90 auth_method 0 2024-03-25T18:27:44.601 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.600+0000 7f24077fe640 10 client.4318.objecter ms_handle_connect 0x7f240c06df90 2024-03-25T18:27:44.605 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.604+0000 7f24077fe640 1 -- 172.21.15.3:0/3701744401 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f2410054590 con 0x7f2420061830 2024-03-25T18:27:44.605 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.604+0000 7f24077fe640 10 client.4318.objecter ms_dispatch 0x7f2420000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:44.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.604+0000 7f24077fe640 3 client.4318.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:44.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.604+0000 7f24077fe640 3 client.4318.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:44.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.604+0000 7f24077fe640 20 client.4318.objecter dump_active .. 0 homeless 2024-03-25T18:27:44.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.605+0000 7f24077fe640 1 -- 172.21.15.3:0/3701744401 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f24100500a0 con 0x7f2420061830 2024-03-25T18:27:44.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.605+0000 7f24077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:44.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.605+0000 7f24077fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:44.840 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.839+0000 7f24255e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:44.840 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.839+0000 7f24255e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:44.840 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.839+0000 7f24255e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:27:44.840 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.839+0000 7f24255e8640 1 -- 172.21.15.3:0/3701744401 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f242013d480 con 0x7f2420061830 2024-03-25T18:27:44.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.860+0000 7f24077fe640 1 -- 172.21.15.3:0/3701744401 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f2410051fd0 con 0x7f2420061830 2024-03-25T18:27:44.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.860+0000 7f24077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:44.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.860+0000 7f24077fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:44.862 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:27:44.862 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:44.864 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.862+0000 7f24255e8640 10 librados: watch_flush enter 2024-03-25T18:27:44.864 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.862+0000 7f24255e8640 10 librados: watch_flush exit 2024-03-25T18:27:44.864 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.862+0000 7f24255e8640 20 client.4318.objecter shutdown clearing up homeless session... 2024-03-25T18:27:44.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.862+0000 7f24255e8640 10 client.4318.objecter successfully canceled tick 2024-03-25T18:27:44.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.862+0000 7f24255e8640 1 -- 172.21.15.3:0/3701744401 >> v1:172.21.15.3:6800/4195140966 conn(0x7f240c06df90 legacy=0x7f240c070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:44.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.862+0000 7f24255e8640 10 monclient: shutdown 2024-03-25T18:27:44.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.862+0000 7f24255e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:44.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.862+0000 7f24255e8640 1 -- 172.21.15.3:0/3701744401 >> v1:172.21.15.16:6789/0 conn(0x7f2420061830 legacy=0x7f2420133730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:44.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.862+0000 7f24255e8640 1 -- 172.21.15.3:0/3701744401 shutdown_connections 2024-03-25T18:27:44.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.862+0000 7f24255e8640 1 -- 172.21.15.3:0/3701744401 >> 172.21.15.3:0/3701744401 conn(0x7f24200b16c0 msgr2=0x7f24201cc150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:44.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.863+0000 7f24255e8640 1 -- 172.21.15.3:0/3701744401 shutdown_connections 2024-03-25T18:27:44.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.863+0000 7f24255e8640 1 -- 172.21.15.3:0/3701744401 wait complete. 2024-03-25T18:27:44.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:44.863+0000 7f24255e8640 1 librados: shutdown 2024-03-25T18:27:44.878 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:51.686 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:51.797 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.796+0000 7f4c3ea06640 10 monclient: get_monmap_and_config 2024-03-25T18:27:51.798 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.796+0000 7f4c3ea06640 10 monclient: build_initial_monmap 2024-03-25T18:27:51.798 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.796+0000 7f4c3ea06640 10 monclient: monmap: 2024-03-25T18:27:51.798 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:27:51.798 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:51.798 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:27:51.797834+0000 2024-03-25T18:27:51.798 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:27:51.797834+0000 2024-03-25T18:27:51.798 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:51.798 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:51.798 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:27:51.798 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:27:51.798 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:27:51.798 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:51.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.797+0000 7f4c3ea06640 1 Processor -- start 2024-03-25T18:27:51.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.797+0000 7f4c3ea06640 1 -- start start 2024-03-25T18:27:51.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.797+0000 7f4c3ea06640 10 monclient: init 2024-03-25T18:27:51.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.798+0000 7f4c3ea06640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:51.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.798+0000 7f4c3ea06640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:27:51.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.798+0000 7f4c3ea06640 10 monclient(hunting): picked mon.noname-c con 0x7f4c380b4e70 addr v1:172.21.15.16:6789/0 2024-03-25T18:27:51.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.798+0000 7f4c3ea06640 10 monclient(hunting): picked mon.noname-b con 0x7f4c380b44b0 addr v1:172.21.15.3:6790/0 2024-03-25T18:27:51.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.798+0000 7f4c3ea06640 10 monclient(hunting): picked mon.noname-a con 0x7f4c380b1b00 addr v1:172.21.15.3:6789/0 2024-03-25T18:27:51.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.798+0000 7f4c3ea06640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c380a96f0 con 0x7f4c380b1b00 2024-03-25T18:27:51.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.798+0000 7f4c3ea06640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c380ab0e0 con 0x7f4c380b4e70 2024-03-25T18:27:51.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.798+0000 7f4c3ea06640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c38145f70 con 0x7f4c380b44b0 2024-03-25T18:27:51.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.798+0000 7f4c3ea06640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:51.801 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.798+0000 7f4c3ea06640 10 monclient(hunting): authenticate will time out at 7526.058888s 2024-03-25T18:27:51.801 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.799+0000 7f4c37fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4c380b4e70 0x7f4c3813f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:49530/0 (socket says 172.21.15.3:49530) 2024-03-25T18:27:51.801 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.799+0000 7f4c37fff640 1 -- 172.21.15.3:0/3329553885 learned_addr learned my addr 172.21.15.3:0/3329553885 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:27:51.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.870+0000 7f4c36ffd640 1 -- 172.21.15.3:0/3329553885 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 497407334 0 0) 0x7f4c380a96f0 con 0x7f4c380b1b00 2024-03-25T18:27:51.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.870+0000 7f4c36ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:51.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.870+0000 7f4c36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:51.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.870+0000 7f4c36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:51.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.870+0000 7f4c36ffd640 10 monclient(hunting): my global_id is 4359 2024-03-25T18:27:51.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.870+0000 7f4c36ffd640 1 -- 172.21.15.3:0/3329553885 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c20003520 con 0x7f4c380b1b00 2024-03-25T18:27:51.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.904+0000 7f4c36ffd640 1 -- 172.21.15.3:0/3329553885 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3569387400 0 0) 0x7f4c20003520 con 0x7f4c380b1b00 2024-03-25T18:27:51.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.905+0000 7f4c36ffd640 1 -- 172.21.15.3:0/3329553885 >> v1:172.21.15.3:6790/0 conn(0x7f4c380b44b0 legacy=0x7f4c38145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:51.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.905+0000 7f4c36ffd640 1 -- 172.21.15.3:0/3329553885 >> v1:172.21.15.16:6789/0 conn(0x7f4c380b4e70 legacy=0x7f4c3813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:51.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.905+0000 7f4c36ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:27:51.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.905+0000 7f4c36ffd640 1 monclient: found mon.noname-a 2024-03-25T18:27:51.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.905+0000 7f4c36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:51.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.905+0000 7f4c36ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:27:51.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.905+0000 7f4c36ffd640 1 -- 172.21.15.3:0/3329553885 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c380a9b80 con 0x7f4c380b1b00 2024-03-25T18:27:51.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.905+0000 7f4c36ffd640 10 monclient: _finish_auth 0 2024-03-25T18:27:51.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.905+0000 7f4c36ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:27:51.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.905+0000 7f4c36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:51.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.905+0000 7f4c36ffd640 1 -- 172.21.15.3:0/3329553885 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4c30001a10 con 0x7f4c380b1b00 2024-03-25T18:27:51.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.905+0000 7f4c3ea06640 5 monclient: authenticate success, global_id 4359 2024-03-25T18:27:51.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.905+0000 7f4c3ea06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:51.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.905+0000 7f4c36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:51.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.905+0000 7f4c36ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:51.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.905+0000 7f4c36ffd640 10 monclient: dump: 2024-03-25T18:27:51.907 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:51.907 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:51.907 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:51.908 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:51.908 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:51.908 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:51.908 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:51.908 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:51.908 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:51.908 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:51.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.905+0000 7f4c3ea06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:51.926 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.925+0000 7f4c36ffd640 1 -- 172.21.15.3:0/3329553885 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c30001b70 con 0x7f4c380b1b00 2024-03-25T18:27:51.926 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.925+0000 7f4c36ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:51.926 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.925+0000 7f4c36ffd640 1 -- 172.21.15.3:0/3329553885 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4c30001f80 con 0x7f4c380b1b00 2024-03-25T18:27:51.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.925+0000 7f4c3ea06640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:51.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.925+0000 7f4c3ea06640 4 set_mon_vals no callback set 2024-03-25T18:27:51.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.925+0000 7f4c36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:51.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.925+0000 7f4c36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:51.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.925+0000 7f4c36ffd640 10 monclient: dump: 2024-03-25T18:27:51.927 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:51.927 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:51.927 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:51.927 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:51.927 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:51.927 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:51.927 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:51.927 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:51.928 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:51.928 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:51.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.926+0000 7f4c3ea06640 10 monclient: shutdown 2024-03-25T18:27:51.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.926+0000 7f4c3ea06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:51.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.926+0000 7f4c3ea06640 1 -- 172.21.15.3:0/3329553885 >> v1:172.21.15.3:6789/0 conn(0x7f4c380b1b00 legacy=0x7f4c3814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:51.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.926+0000 7f4c3ea06640 1 -- 172.21.15.3:0/3329553885 shutdown_connections 2024-03-25T18:27:51.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.926+0000 7f4c3ea06640 1 -- 172.21.15.3:0/3329553885 >> 172.21.15.3:0/3329553885 conn(0x7f4c380617a0 msgr2=0x7f4c38061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:51.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.926+0000 7f4c3ea06640 1 -- 172.21.15.3:0/3329553885 shutdown_connections 2024-03-25T18:27:51.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.926+0000 7f4c3ea06640 1 -- 172.21.15.3:0/3329553885 wait complete. 2024-03-25T18:27:51.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.927+0000 7f4c3ea06640 10 monclient: build_initial_monmap 2024-03-25T18:27:51.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.927+0000 7f4c3ea06640 10 monclient: monmap: 2024-03-25T18:27:51.931 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:27:51.931 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:51.931 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:27:51.931 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:27:51.931 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:51.931 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:27:51.931 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:27:51.931 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:27:51.931 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:27:51.931 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:51.931 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.927+0000 7f4c3ea06640 1 librados: starting msgr at 2024-03-25T18:27:51.931 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.927+0000 7f4c3ea06640 1 librados: starting objecter 2024-03-25T18:27:51.931 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.927+0000 7f4c3ea06640 1 Processor -- start 2024-03-25T18:27:51.931 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.928+0000 7f4c3ea06640 1 -- start start 2024-03-25T18:27:51.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.928+0000 7f4c3ea06640 1 librados: setting wanted keys 2024-03-25T18:27:51.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.928+0000 7f4c3ea06640 1 librados: calling monclient init 2024-03-25T18:27:51.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.928+0000 7f4c3ea06640 10 monclient: init 2024-03-25T18:27:51.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.928+0000 7f4c3ea06640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:51.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.928+0000 7f4c3ea06640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:27:51.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.928+0000 7f4c3ea06640 10 monclient(hunting): picked mon.noname-a con 0x7f4c380b1b00 addr v1:172.21.15.3:6789/0 2024-03-25T18:27:51.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.928+0000 7f4c3ea06640 10 monclient(hunting): picked mon.noname-c con 0x7f4c380b44b0 addr v1:172.21.15.3:6790/0 2024-03-25T18:27:51.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.928+0000 7f4c3ea06640 10 monclient(hunting): picked mon.noname-b con 0x7f4c380b4e70 addr v1:172.21.15.16:6789/0 2024-03-25T18:27:51.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.928+0000 7f4c3ea06640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c381d76e0 con 0x7f4c380b1b00 2024-03-25T18:27:51.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.928+0000 7f4c3ea06640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c38131c10 con 0x7f4c380b4e70 2024-03-25T18:27:51.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.928+0000 7f4c3ea06640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c3812f540 con 0x7f4c380b44b0 2024-03-25T18:27:51.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.928+0000 7f4c3ea06640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:51.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.928+0000 7f4c3ea06640 10 monclient(hunting): authenticate will time out at 7526.189214s 2024-03-25T18:27:51.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.937+0000 7f4c3cfa3640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4c380b4e70 0x7f4c381d7b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:49576/0 (socket says 172.21.15.3:49576) 2024-03-25T18:27:51.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.937+0000 7f4c3cfa3640 1 -- 172.21.15.3:0/3722015494 learned_addr learned my addr 172.21.15.3:0/3722015494 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:27:51.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.937+0000 7f4c34ff9640 10 client.?.objecter ms_handle_connect 0x7f4c380b4e70 2024-03-25T18:27:51.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.937+0000 7f4c34ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:51.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.938+0000 7f4c34ff9640 10 client.?.objecter ms_handle_connect 0x7f4c380b44b0 2024-03-25T18:27:51.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.938+0000 7f4c34ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:51.968 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.967+0000 7f4c34ff9640 10 client.?.objecter ms_handle_connect 0x7f4c380b1b00 2024-03-25T18:27:51.969 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:51.967+0000 7f4c34ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:52.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.029+0000 7f4c34ff9640 1 -- 172.21.15.3:0/3722015494 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886830278 0 0) 0x7f4c38131c10 con 0x7f4c380b4e70 2024-03-25T18:27:52.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.029+0000 7f4c34ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:52.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.029+0000 7f4c34ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:52.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.029+0000 7f4c34ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:52.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.029+0000 7f4c34ff9640 10 monclient(hunting): my global_id is 4324 2024-03-25T18:27:52.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.029+0000 7f4c34ff9640 1 -- 172.21.15.3:0/3722015494 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c24004eb0 con 0x7f4c380b4e70 2024-03-25T18:27:52.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.030+0000 7f4c34ff9640 1 -- 172.21.15.3:0/3722015494 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1699027648 0 0) 0x7f4c381d76e0 con 0x7f4c380b1b00 2024-03-25T18:27:52.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.030+0000 7f4c34ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:52.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.030+0000 7f4c34ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:52.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.030+0000 7f4c34ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:52.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.030+0000 7f4c34ff9640 10 monclient(hunting): my global_id is 4362 2024-03-25T18:27:52.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.030+0000 7f4c34ff9640 1 -- 172.21.15.3:0/3722015494 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c38131c10 con 0x7f4c380b1b00 2024-03-25T18:27:52.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.046+0000 7f4c34ff9640 1 -- 172.21.15.3:0/3722015494 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3557768503 0 0) 0x7f4c24004eb0 con 0x7f4c380b4e70 2024-03-25T18:27:52.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.046+0000 7f4c34ff9640 1 -- 172.21.15.3:0/3722015494 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c381d76e0 con 0x7f4c380b4e70 2024-03-25T18:27:52.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.046+0000 7f4c34ff9640 1 -- 172.21.15.3:0/3722015494 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4c30001f80 con 0x7f4c380b4e70 2024-03-25T18:27:52.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.046+0000 7f4c34ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:52.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.046+0000 7f4c34ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:52.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.046+0000 7f4c34ff9640 10 monclient(hunting): dump: 2024-03-25T18:27:52.047 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:52.047 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:52.048 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:52.048 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:52.048 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:52.048 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:52.048 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:52.048 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:52.048 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:52.048 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:52.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.046+0000 7f4c34ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:52.053 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.052+0000 7f4c34ff9640 1 -- 172.21.15.3:0/3722015494 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1658959830 0 0) 0x7f4c38131c10 con 0x7f4c380b1b00 2024-03-25T18:27:52.053 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.052+0000 7f4c34ff9640 1 -- 172.21.15.3:0/3722015494 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c24004eb0 con 0x7f4c380b1b00 2024-03-25T18:27:52.053 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.052+0000 7f4c34ff9640 1 -- 172.21.15.3:0/3722015494 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4c28001800 con 0x7f4c380b1b00 2024-03-25T18:27:52.053 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.052+0000 7f4c34ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:52.053 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.052+0000 7f4c34ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:52.053 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.052+0000 7f4c34ff9640 10 monclient(hunting): dump: 2024-03-25T18:27:52.054 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:52.054 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:52.054 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:52.054 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:52.054 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:52.054 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:52.054 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:52.054 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:52.054 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:52.054 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:52.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.069+0000 7f4c34ff9640 1 -- 172.21.15.3:0/3722015494 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1717784679 0 0) 0x7f4c24004eb0 con 0x7f4c380b1b00 2024-03-25T18:27:52.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.069+0000 7f4c34ff9640 1 -- 172.21.15.3:0/3722015494 >> v1:172.21.15.3:6790/0 conn(0x7f4c380b44b0 legacy=0x7f4c381db030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:52.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.069+0000 7f4c34ff9640 1 -- 172.21.15.3:0/3722015494 >> v1:172.21.15.16:6789/0 conn(0x7f4c380b4e70 legacy=0x7f4c381d7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:52.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.069+0000 7f4c34ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:27:52.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.069+0000 7f4c34ff9640 1 monclient: found mon.a 2024-03-25T18:27:52.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.069+0000 7f4c34ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:52.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.070+0000 7f4c34ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:27:52.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.070+0000 7f4c34ff9640 1 -- 172.21.15.3:0/3722015494 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c3812f6d0 con 0x7f4c380b1b00 2024-03-25T18:27:52.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.070+0000 7f4c34ff9640 10 monclient: _finish_auth 0 2024-03-25T18:27:52.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.070+0000 7f4c34ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:27:52.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.070+0000 7f4c34ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:52.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.070+0000 7f4c3ea06640 5 monclient: authenticate success, global_id 4362 2024-03-25T18:27:52.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.070+0000 7f4c3ea06640 10 monclient: _renew_subs 2024-03-25T18:27:52.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.070+0000 7f4c3ea06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:27:52.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.070+0000 7f4c3ea06640 1 -- 172.21.15.3:0/3722015494 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c381307d0 con 0x7f4c380b1b00 2024-03-25T18:27:52.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.070+0000 7f4c3ea06640 10 client.4362.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:52.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.070+0000 7f4c3ea06640 10 monclient: _renew_subs 2024-03-25T18:27:52.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.070+0000 7f4c3ea06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:27:52.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.070+0000 7f4c3ea06640 1 -- 172.21.15.3:0/3722015494 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c381db920 con 0x7f4c380b1b00 2024-03-25T18:27:52.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.070+0000 7f4c3ea06640 1 librados: init done 2024-03-25T18:27:52.073 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.071+0000 7f4c3ea06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:52.073 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.071+0000 7f4c3ea06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:52.073 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.071+0000 7f4c3ea06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:27:52.073 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.071+0000 7f4c3ea06640 1 -- 172.21.15.3:0/3722015494 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c3804f730 con 0x7f4c380b1b00 2024-03-25T18:27:52.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.393+0000 7f4c34ff9640 1 -- 172.21.15.3:0/3722015494 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c28001c10 con 0x7f4c380b1b00 2024-03-25T18:27:52.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.393+0000 7f4c34ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:52.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.393+0000 7f4c34ff9640 1 -- 172.21.15.3:0/3722015494 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4c28004980 con 0x7f4c380b1b00 2024-03-25T18:27:52.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.393+0000 7f4c34ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:52.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.394+0000 7f4c34ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:52.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.394+0000 7f4c34ff9640 10 monclient: dump: 2024-03-25T18:27:52.395 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:52.395 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:52.395 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:52.396 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:52.396 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:52.396 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:52.396 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:52.396 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:52.396 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:52.396 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:52.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.393+0000 7f4c367fc640 4 set_mon_vals no callback set 2024-03-25T18:27:52.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.395+0000 7f4c34ff9640 1 -- 172.21.15.3:0/3722015494 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f4c28004ba0 con 0x7f4c380b1b00 2024-03-25T18:27:52.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.395+0000 7f4c377fe640 10 monclient: get_auth_request con 0x7f4c2406de90 auth_method 0 2024-03-25T18:27:52.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.395+0000 7f4c34ff9640 10 client.4362.objecter ms_handle_connect 0x7f4c2406de90 2024-03-25T18:27:52.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.401+0000 7f4c34ff9640 1 -- 172.21.15.3:0/3722015494 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f4c28054660 con 0x7f4c380b1b00 2024-03-25T18:27:52.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.401+0000 7f4c34ff9640 10 client.4362.objecter ms_dispatch 0x7f4c38000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:52.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.401+0000 7f4c34ff9640 3 client.4362.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:52.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.401+0000 7f4c34ff9640 3 client.4362.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:52.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.401+0000 7f4c34ff9640 20 client.4362.objecter dump_active .. 0 homeless 2024-03-25T18:27:52.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.401+0000 7f4c34ff9640 1 -- 172.21.15.3:0/3722015494 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4c28050170 con 0x7f4c380b1b00 2024-03-25T18:27:52.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.401+0000 7f4c34ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:52.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.401+0000 7f4c34ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:52.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.653+0000 7f4c3ea06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:52.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.653+0000 7f4c3ea06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:52.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.653+0000 7f4c3ea06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:27:52.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.653+0000 7f4c3ea06640 1 -- 172.21.15.3:0/3722015494 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c380aa090 con 0x7f4c380b1b00 2024-03-25T18:27:52.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.681+0000 7f4c34ff9640 1 -- 172.21.15.3:0/3722015494 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f4c280520a0 con 0x7f4c380b1b00 2024-03-25T18:27:52.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.681+0000 7f4c34ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:52.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.681+0000 7f4c34ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:52.682 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:27:52.683 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:52.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.683+0000 7f4c3ea06640 10 librados: watch_flush enter 2024-03-25T18:27:52.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.683+0000 7f4c3ea06640 10 librados: watch_flush exit 2024-03-25T18:27:52.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.683+0000 7f4c3ea06640 20 client.4362.objecter shutdown clearing up homeless session... 2024-03-25T18:27:52.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.683+0000 7f4c3ea06640 10 client.4362.objecter successfully canceled tick 2024-03-25T18:27:52.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.683+0000 7f4c3ea06640 1 -- 172.21.15.3:0/3722015494 >> v1:172.21.15.3:6800/4195140966 conn(0x7f4c2406de90 legacy=0x7f4c24070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:52.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.683+0000 7f4c3ea06640 10 monclient: shutdown 2024-03-25T18:27:52.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.683+0000 7f4c3ea06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:52.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.683+0000 7f4c3ea06640 1 -- 172.21.15.3:0/3722015494 >> v1:172.21.15.3:6789/0 conn(0x7f4c380b1b00 legacy=0x7f4c381300c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:52.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.683+0000 7f4c3ea06640 1 -- 172.21.15.3:0/3722015494 shutdown_connections 2024-03-25T18:27:52.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.683+0000 7f4c3ea06640 1 -- 172.21.15.3:0/3722015494 >> 172.21.15.3:0/3722015494 conn(0x7f4c380617a0 msgr2=0x7f4c380b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:52.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.683+0000 7f4c3ea06640 1 -- 172.21.15.3:0/3722015494 shutdown_connections 2024-03-25T18:27:52.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.684+0000 7f4c3ea06640 1 -- 172.21.15.3:0/3722015494 wait complete. 2024-03-25T18:27:52.686 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:52.684+0000 7f4c3ea06640 1 librados: shutdown 2024-03-25T18:27:52.696 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:59.504 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:59.621 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.619+0000 7f4159855640 10 monclient: get_monmap_and_config 2024-03-25T18:27:59.621 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.619+0000 7f4159855640 10 monclient: build_initial_monmap 2024-03-25T18:27:59.621 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.619+0000 7f4159855640 10 monclient: monmap: 2024-03-25T18:27:59.621 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:27:59.621 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:59.621 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:27:59.620641+0000 2024-03-25T18:27:59.621 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:27:59.620641+0000 2024-03-25T18:27:59.621 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:59.621 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:59.621 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:27:59.622 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:27:59.622 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:27:59.622 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:59.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.620+0000 7f4159855640 1 Processor -- start 2024-03-25T18:27:59.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.620+0000 7f4159855640 1 -- start start 2024-03-25T18:27:59.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.620+0000 7f4159855640 10 monclient: init 2024-03-25T18:27:59.623 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.621+0000 7f4159855640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:59.623 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.621+0000 7f4159855640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:27:59.623 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.621+0000 7f4159855640 10 monclient(hunting): picked mon.noname-b con 0x7f41540b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:27:59.623 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.621+0000 7f4159855640 10 monclient(hunting): picked mon.noname-c con 0x7f41540b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:27:59.623 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.621+0000 7f4159855640 10 monclient(hunting): picked mon.noname-a con 0x7f415413c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:27:59.623 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.621+0000 7f4159855640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41540ab0e0 con 0x7f415413c480 2024-03-25T18:27:59.623 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.621+0000 7f4159855640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41540ab3c0 con 0x7f41540b4670 2024-03-25T18:27:59.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.621+0000 7f4159855640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41540a9f30 con 0x7f41540b4ff0 2024-03-25T18:27:59.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.621+0000 7f4159855640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:59.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.621+0000 7f4159855640 10 monclient(hunting): authenticate will time out at 7533.881593s 2024-03-25T18:27:59.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.628+0000 7f4152ffd640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f41540b4ff0 0x7f415413bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:47202/0 (socket says 172.21.15.3:47202) 2024-03-25T18:27:59.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.628+0000 7f4152ffd640 1 -- 172.21.15.3:0/2159493819 learned_addr learned my addr 172.21.15.3:0/2159493819 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:27:59.644 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.643+0000 7f4151ffb640 1 -- 172.21.15.3:0/2159493819 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3468731848 0 0) 0x7f41540a9f30 con 0x7f41540b4ff0 2024-03-25T18:27:59.644 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.643+0000 7f4151ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:27:59.644 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.643+0000 7f4151ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:59.644 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.643+0000 7f4151ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:59.644 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.643+0000 7f4151ffb640 10 monclient(hunting): my global_id is 4334 2024-03-25T18:27:59.644 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.643+0000 7f4151ffb640 1 -- 172.21.15.3:0/2159493819 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f413c003520 con 0x7f41540b4ff0 2024-03-25T18:27:59.648 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.647+0000 7f4151ffb640 1 -- 172.21.15.3:0/2159493819 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3459561120 0 0) 0x7f41540ab3c0 con 0x7f41540b4670 2024-03-25T18:27:59.648 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.647+0000 7f4151ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:27:59.648 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.647+0000 7f4151ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:59.648 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.647+0000 7f4151ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:59.648 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.647+0000 7f4151ffb640 10 monclient(hunting): my global_id is 4327 2024-03-25T18:27:59.648 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.647+0000 7f4151ffb640 1 -- 172.21.15.3:0/2159493819 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41540a9f30 con 0x7f41540b4670 2024-03-25T18:27:59.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.656+0000 7f4151ffb640 1 -- 172.21.15.3:0/2159493819 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 28432429 0 0) 0x7f41540a9f30 con 0x7f41540b4670 2024-03-25T18:27:59.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.657+0000 7f4151ffb640 1 -- 172.21.15.3:0/2159493819 >> v1:172.21.15.3:6790/0 conn(0x7f41540b4ff0 legacy=0x7f415413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:59.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.657+0000 7f4151ffb640 1 -- 172.21.15.3:0/2159493819 >> v1:172.21.15.3:6789/0 conn(0x7f415413c480 legacy=0x7f415414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:59.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.657+0000 7f4151ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:27:59.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.657+0000 7f4151ffb640 1 monclient: found mon.noname-c 2024-03-25T18:27:59.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.657+0000 7f4151ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:59.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.657+0000 7f4151ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:27:59.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.657+0000 7f4151ffb640 1 -- 172.21.15.3:0/2159493819 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41540a9b70 con 0x7f41540b4670 2024-03-25T18:27:59.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.657+0000 7f4151ffb640 10 monclient: _finish_auth 0 2024-03-25T18:27:59.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.657+0000 7f4151ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:27:59.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.657+0000 7f4151ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:59.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.657+0000 7f4151ffb640 1 -- 172.21.15.3:0/2159493819 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4148001670 con 0x7f41540b4670 2024-03-25T18:27:59.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.657+0000 7f4151ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:59.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.657+0000 7f4151ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:59.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.657+0000 7f4151ffb640 10 monclient: dump: 2024-03-25T18:27:59.659 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:59.659 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:59.659 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:59.659 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:59.659 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:59.659 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:59.660 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:59.660 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:59.660 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:59.660 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:59.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.657+0000 7f4159855640 5 monclient: authenticate success, global_id 4327 2024-03-25T18:27:59.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.657+0000 7f4159855640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:59.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.680+0000 7f4151ffb640 1 -- 172.21.15.3:0/2159493819 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41480017d0 con 0x7f41540b4670 2024-03-25T18:27:59.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.680+0000 7f4151ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:59.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.680+0000 7f4151ffb640 1 -- 172.21.15.3:0/2159493819 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4148001b60 con 0x7f41540b4670 2024-03-25T18:27:59.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.680+0000 7f4151ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:59.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.680+0000 7f4151ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:59.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.680+0000 7f4151ffb640 10 monclient: dump: 2024-03-25T18:27:59.682 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:59.682 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:59.682 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:59.682 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:59.683 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:59.683 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:59.683 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:59.683 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:59.683 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:59.683 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:59.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.680+0000 7f4159855640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:59.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.680+0000 7f4159855640 4 set_mon_vals no callback set 2024-03-25T18:27:59.684 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.681+0000 7f4159855640 10 monclient: shutdown 2024-03-25T18:27:59.684 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.681+0000 7f4159855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:59.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.681+0000 7f4159855640 1 -- 172.21.15.3:0/2159493819 >> v1:172.21.15.16:6789/0 conn(0x7f41540b4670 legacy=0x7f41541425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:59.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.681+0000 7f4159855640 1 -- 172.21.15.3:0/2159493819 shutdown_connections 2024-03-25T18:27:59.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.681+0000 7f4159855640 1 -- 172.21.15.3:0/2159493819 >> 172.21.15.3:0/2159493819 conn(0x7f41540617a0 msgr2=0x7f4154061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:59.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.681+0000 7f4159855640 1 -- 172.21.15.3:0/2159493819 shutdown_connections 2024-03-25T18:27:59.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.681+0000 7f4159855640 1 -- 172.21.15.3:0/2159493819 wait complete. 2024-03-25T18:27:59.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.682+0000 7f4159855640 10 monclient: build_initial_monmap 2024-03-25T18:27:59.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.682+0000 7f4159855640 10 monclient: monmap: 2024-03-25T18:27:59.685 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:27:59.685 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:59.685 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:27:59.685 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:27:59.686 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:59.686 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:27:59.686 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:27:59.686 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:27:59.686 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:27:59.686 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:59.686 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.683+0000 7f4159855640 1 librados: starting msgr at 2024-03-25T18:27:59.686 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.683+0000 7f4159855640 1 librados: starting objecter 2024-03-25T18:27:59.686 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.683+0000 7f4159855640 1 Processor -- start 2024-03-25T18:27:59.686 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.683+0000 7f4159855640 1 -- start start 2024-03-25T18:27:59.686 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.683+0000 7f4159855640 1 librados: setting wanted keys 2024-03-25T18:27:59.686 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.683+0000 7f4159855640 1 librados: calling monclient init 2024-03-25T18:27:59.686 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.683+0000 7f4159855640 10 monclient: init 2024-03-25T18:27:59.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.684+0000 7f4159855640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:59.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.684+0000 7f4159855640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:27:59.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.684+0000 7f4159855640 10 monclient(hunting): picked mon.noname-a con 0x7f41540b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:27:59.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.684+0000 7f4159855640 10 monclient(hunting): picked mon.noname-c con 0x7f41540b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:27:59.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.684+0000 7f4159855640 10 monclient(hunting): picked mon.noname-b con 0x7f415413c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:27:59.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.684+0000 7f4159855640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415413c8f0 con 0x7f41540b4670 2024-03-25T18:27:59.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.684+0000 7f4159855640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415413c2b0 con 0x7f415413c480 2024-03-25T18:27:59.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.684+0000 7f4159855640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4154130ad0 con 0x7f41540b4ff0 2024-03-25T18:27:59.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.684+0000 7f4159855640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:59.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.684+0000 7f4159855640 10 monclient(hunting): authenticate will time out at 7533.945057s 2024-03-25T18:27:59.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.686+0000 7f41527fc640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f41540b4ff0 0x7f415412fcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:47230/0 (socket says 172.21.15.3:47230) 2024-03-25T18:27:59.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.686+0000 7f41527fc640 1 -- 172.21.15.3:0/3327736236 learned_addr learned my addr 172.21.15.3:0/3327736236 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:27:59.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.687+0000 7f413bfff640 10 client.?.objecter ms_handle_connect 0x7f41540b4ff0 2024-03-25T18:27:59.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.687+0000 7f413bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:59.702 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.701+0000 7f413bfff640 10 client.?.objecter ms_handle_connect 0x7f415413c480 2024-03-25T18:27:59.702 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.701+0000 7f413bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:59.705 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.704+0000 7f413bfff640 10 client.?.objecter ms_handle_connect 0x7f41540b4670 2024-03-25T18:27:59.705 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.704+0000 7f413bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:59.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.768+0000 7f413bfff640 1 -- 172.21.15.3:0/3327736236 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 765502051 0 0) 0x7f415413c8f0 con 0x7f41540b4670 2024-03-25T18:27:59.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.768+0000 7f413bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:59.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.768+0000 7f413bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:59.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.768+0000 7f413bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:59.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.768+0000 7f413bfff640 10 monclient(hunting): my global_id is 4368 2024-03-25T18:27:59.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.768+0000 7f413bfff640 1 -- 172.21.15.3:0/3327736236 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4140004eb0 con 0x7f41540b4670 2024-03-25T18:27:59.778 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.777+0000 7f413bfff640 1 -- 172.21.15.3:0/3327736236 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1824005414 0 0) 0x7f4140004eb0 con 0x7f41540b4670 2024-03-25T18:27:59.778 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.777+0000 7f413bfff640 1 -- 172.21.15.3:0/3327736236 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f415413c8f0 con 0x7f41540b4670 2024-03-25T18:27:59.778 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.777+0000 7f413bfff640 1 -- 172.21.15.3:0/3327736236 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4144001950 con 0x7f41540b4670 2024-03-25T18:27:59.778 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.777+0000 7f413bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:59.778 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.777+0000 7f413bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:59.779 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.777+0000 7f413bfff640 10 monclient(hunting): dump: 2024-03-25T18:27:59.779 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:27:59.779 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:27:59.779 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:59.779 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:27:59.779 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:59.779 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:27:59.779 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:27:59.780 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:27:59.780 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:27:59.780 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:27:59.780 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.777+0000 7f413bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:59.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.784+0000 7f413bfff640 1 -- 172.21.15.3:0/3327736236 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3774920127 0 0) 0x7f415413c8f0 con 0x7f41540b4670 2024-03-25T18:27:59.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.784+0000 7f413bfff640 1 -- 172.21.15.3:0/3327736236 >> v1:172.21.15.3:6790/0 conn(0x7f41540b4ff0 legacy=0x7f415412fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:59.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.784+0000 7f413bfff640 1 -- 172.21.15.3:0/3327736236 >> v1:172.21.15.16:6789/0 conn(0x7f415413c480 legacy=0x7f41541303c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:59.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.784+0000 7f413bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:27:59.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.784+0000 7f413bfff640 1 monclient: found mon.a 2024-03-25T18:27:59.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.784+0000 7f413bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:59.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.784+0000 7f413bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:27:59.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.784+0000 7f413bfff640 1 -- 172.21.15.3:0/3327736236 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415413e8f0 con 0x7f41540b4670 2024-03-25T18:27:59.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.784+0000 7f413bfff640 10 monclient: _finish_auth 0 2024-03-25T18:27:59.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.784+0000 7f413bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:27:59.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.784+0000 7f413bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:59.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.784+0000 7f4159855640 5 monclient: authenticate success, global_id 4368 2024-03-25T18:27:59.787 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.784+0000 7f4159855640 10 monclient: _renew_subs 2024-03-25T18:27:59.787 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.784+0000 7f4159855640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:27:59.787 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.784+0000 7f4159855640 1 -- 172.21.15.3:0/3327736236 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f415413d790 con 0x7f41540b4670 2024-03-25T18:27:59.787 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.784+0000 7f4159855640 10 client.4368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:59.787 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.784+0000 7f4159855640 10 monclient: _renew_subs 2024-03-25T18:27:59.787 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.784+0000 7f4159855640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:27:59.787 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.784+0000 7f4159855640 1 -- 172.21.15.3:0/3327736236 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f415413db20 con 0x7f41540b4670 2024-03-25T18:27:59.787 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.784+0000 7f4159855640 1 librados: init done 2024-03-25T18:27:59.788 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.785+0000 7f4159855640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:59.788 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.785+0000 7f4159855640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:59.788 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.785+0000 7f4159855640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:27:59.788 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:27:59.785+0000 7f4159855640 1 -- 172.21.15.3:0/3327736236 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f415413cc60 con 0x7f41540b4670 2024-03-25T18:28:00.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.102+0000 7f413bfff640 1 -- 172.21.15.3:0/3327736236 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4144001d60 con 0x7f41540b4670 2024-03-25T18:28:00.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.102+0000 7f413bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:00.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.102+0000 7f413bfff640 1 -- 172.21.15.3:0/3327736236 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f41440049c0 con 0x7f41540b4670 2024-03-25T18:28:00.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.102+0000 7f413bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:00.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.102+0000 7f413bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:00.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.102+0000 7f413bfff640 10 monclient: dump: 2024-03-25T18:28:00.104 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:00.104 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:00.104 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:00.104 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:00.104 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:00.104 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:00.104 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:00.104 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:00.104 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:00.104 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:00.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.102+0000 7f41517fa640 4 set_mon_vals no callback set 2024-03-25T18:28:00.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.103+0000 7f413bfff640 1 -- 172.21.15.3:0/3327736236 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f4144004be0 con 0x7f41540b4670 2024-03-25T18:28:00.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.104+0000 7f41527fc640 10 monclient: get_auth_request con 0x7f414006dd00 auth_method 0 2024-03-25T18:28:00.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.104+0000 7f413bfff640 10 client.4368.objecter ms_handle_connect 0x7f414006dd00 2024-03-25T18:28:00.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.113+0000 7f413bfff640 1 -- 172.21.15.3:0/3327736236 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f4144054750 con 0x7f41540b4670 2024-03-25T18:28:00.115 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.113+0000 7f413bfff640 10 client.4368.objecter ms_dispatch 0x7f4154000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:00.115 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.113+0000 7f413bfff640 3 client.4368.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:00.115 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.113+0000 7f413bfff640 3 client.4368.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:00.115 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.113+0000 7f413bfff640 20 client.4368.objecter dump_active .. 0 homeless 2024-03-25T18:28:00.115 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.113+0000 7f413bfff640 1 -- 172.21.15.3:0/3327736236 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4144050090 con 0x7f41540b4670 2024-03-25T18:28:00.115 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.113+0000 7f413bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:00.115 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.113+0000 7f413bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:00.366 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.365+0000 7f4159855640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:00.366 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.365+0000 7f4159855640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:00.366 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.365+0000 7f4159855640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:28:00.366 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.365+0000 7f4159855640 1 -- 172.21.15.3:0/3327736236 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f415406dbd0 con 0x7f41540b4670 2024-03-25T18:28:00.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.387+0000 7f413bfff640 1 -- 172.21.15.3:0/3327736236 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f4144052190 con 0x7f41540b4670 2024-03-25T18:28:00.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.387+0000 7f413bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:00.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.387+0000 7f413bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:00.389 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:28:00.389 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:00.392 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.390+0000 7f4159855640 10 librados: watch_flush enter 2024-03-25T18:28:00.392 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.390+0000 7f4159855640 10 librados: watch_flush exit 2024-03-25T18:28:00.392 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.390+0000 7f4159855640 20 client.4368.objecter shutdown clearing up homeless session... 2024-03-25T18:28:00.392 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.390+0000 7f4159855640 10 client.4368.objecter successfully canceled tick 2024-03-25T18:28:00.393 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.390+0000 7f4159855640 1 -- 172.21.15.3:0/3327736236 >> v1:172.21.15.3:6800/4195140966 conn(0x7f414006dd00 legacy=0x7f41400701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:00.393 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.390+0000 7f4159855640 10 monclient: shutdown 2024-03-25T18:28:00.393 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.390+0000 7f4159855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:00.393 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.390+0000 7f4159855640 1 -- 172.21.15.3:0/3327736236 >> v1:172.21.15.3:6789/0 conn(0x7f41540b4670 legacy=0x7f415413e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:00.393 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.390+0000 7f4159855640 1 -- 172.21.15.3:0/3327736236 shutdown_connections 2024-03-25T18:28:00.393 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.390+0000 7f4159855640 1 -- 172.21.15.3:0/3327736236 >> 172.21.15.3:0/3327736236 conn(0x7f41540617a0 msgr2=0x7f41540b2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:00.393 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.390+0000 7f4159855640 1 -- 172.21.15.3:0/3327736236 shutdown_connections 2024-03-25T18:28:00.393 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.391+0000 7f4159855640 1 -- 172.21.15.3:0/3327736236 wait complete. 2024-03-25T18:28:00.393 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:00.391+0000 7f4159855640 1 librados: shutdown 2024-03-25T18:28:00.403 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:01.196 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:03:14.925 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:01.256 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:03:14.985 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:01.281 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:03:15.010 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:01.300 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:03:14.992 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:01.306 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:03:14.997 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:01.311 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:03:15.040 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:01.319 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:03:15.048 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:01.335 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:03:15.027 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:01.357 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:03:15.086 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:07.211 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:07.328 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.326+0000 7ffb34de8640 10 monclient: get_monmap_and_config 2024-03-25T18:28:07.328 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.326+0000 7ffb34de8640 10 monclient: build_initial_monmap 2024-03-25T18:28:07.328 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.326+0000 7ffb34de8640 10 monclient: monmap: 2024-03-25T18:28:07.328 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:28:07.328 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:07.328 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:28:07.327892+0000 2024-03-25T18:28:07.328 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:28:07.327892+0000 2024-03-25T18:28:07.328 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:07.329 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:07.329 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:28:07.329 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:28:07.329 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:28:07.329 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:07.330 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.327+0000 7ffb34de8640 1 Processor -- start 2024-03-25T18:28:07.330 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.327+0000 7ffb34de8640 1 -- start start 2024-03-25T18:28:07.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.327+0000 7ffb34de8640 10 monclient: init 2024-03-25T18:28:07.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.328+0000 7ffb34de8640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:07.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.328+0000 7ffb34de8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:28:07.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.328+0000 7ffb34de8640 10 monclient(hunting): picked mon.noname-c con 0x7ffb300b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:28:07.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.328+0000 7ffb34de8640 10 monclient(hunting): picked mon.noname-b con 0x7ffb300b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:28:07.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.328+0000 7ffb34de8640 10 monclient(hunting): picked mon.noname-a con 0x7ffb3013c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:28:07.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.328+0000 7ffb34de8640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb300ab0e0 con 0x7ffb3013c480 2024-03-25T18:28:07.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.328+0000 7ffb34de8640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb300ab3c0 con 0x7ffb300b4ff0 2024-03-25T18:28:07.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.328+0000 7ffb34de8640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb300a9f30 con 0x7ffb300b4670 2024-03-25T18:28:07.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.328+0000 7ffb34de8640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:07.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.328+0000 7ffb34de8640 10 monclient(hunting): authenticate will time out at 7541.588762s 2024-03-25T18:28:07.334 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.333+0000 7ffb2e59c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ffb300b4ff0 0x7ffb3013bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:40936/0 (socket says 172.21.15.3:40936) 2024-03-25T18:28:07.334 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.333+0000 7ffb2e59c640 1 -- 172.21.15.3:0/3299533495 learned_addr learned my addr 172.21.15.3:0/3299533495 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:28:07.348 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.347+0000 7ffb2dd9b640 1 -- 172.21.15.3:0/3299533495 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2978506215 0 0) 0x7ffb300a9f30 con 0x7ffb300b4670 2024-03-25T18:28:07.348 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.347+0000 7ffb2dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:28:07.348 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.347+0000 7ffb2dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:07.348 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.347+0000 7ffb2dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:07.348 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.347+0000 7ffb2dd9b640 10 monclient(hunting): my global_id is 4340 2024-03-25T18:28:07.348 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.347+0000 7ffb2dd9b640 1 -- 172.21.15.3:0/3299533495 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb1c003520 con 0x7ffb300b4670 2024-03-25T18:28:07.349 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.348+0000 7ffb2dd9b640 1 -- 172.21.15.3:0/3299533495 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027196704 0 0) 0x7ffb300ab3c0 con 0x7ffb300b4ff0 2024-03-25T18:28:07.350 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.348+0000 7ffb2dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:28:07.350 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.348+0000 7ffb2dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:07.350 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.348+0000 7ffb2dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:07.350 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.348+0000 7ffb2dd9b640 10 monclient(hunting): my global_id is 4333 2024-03-25T18:28:07.350 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.348+0000 7ffb2dd9b640 1 -- 172.21.15.3:0/3299533495 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb300a9f30 con 0x7ffb300b4ff0 2024-03-25T18:28:07.351 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.350+0000 7ffb2dd9b640 1 -- 172.21.15.3:0/3299533495 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 991846612 0 0) 0x7ffb300ab0e0 con 0x7ffb3013c480 2024-03-25T18:28:07.351 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.350+0000 7ffb2dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:28:07.351 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.350+0000 7ffb2dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:07.351 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.350+0000 7ffb2dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:07.351 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.350+0000 7ffb2dd9b640 10 monclient(hunting): my global_id is 4371 2024-03-25T18:28:07.352 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.350+0000 7ffb2dd9b640 1 -- 172.21.15.3:0/3299533495 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb300ab3c0 con 0x7ffb3013c480 2024-03-25T18:28:07.363 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.362+0000 7ffb2dd9b640 1 -- 172.21.15.3:0/3299533495 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3665371525 0 0) 0x7ffb1c003520 con 0x7ffb300b4670 2024-03-25T18:28:07.363 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.362+0000 7ffb2dd9b640 1 -- 172.21.15.3:0/3299533495 >> v1:172.21.15.16:6789/0 conn(0x7ffb300b4ff0 legacy=0x7ffb3013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:07.363 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.362+0000 7ffb2dd9b640 1 -- 172.21.15.3:0/3299533495 >> v1:172.21.15.3:6789/0 conn(0x7ffb3013c480 legacy=0x7ffb3014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:07.364 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.362+0000 7ffb2dd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:28:07.364 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.362+0000 7ffb2dd9b640 1 monclient: found mon.noname-b 2024-03-25T18:28:07.364 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.362+0000 7ffb2dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:07.364 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.362+0000 7ffb2dd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-25T18:28:07.364 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.362+0000 7ffb2dd9b640 1 -- 172.21.15.3:0/3299533495 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb300a9b70 con 0x7ffb300b4670 2024-03-25T18:28:07.364 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.362+0000 7ffb2dd9b640 10 monclient: _finish_auth 0 2024-03-25T18:28:07.364 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.362+0000 7ffb2dd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:28:07.364 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.362+0000 7ffb2dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:07.364 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.362+0000 7ffb2dd9b640 1 -- 172.21.15.3:0/3299533495 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7ffb18001690 con 0x7ffb300b4670 2024-03-25T18:28:07.364 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.362+0000 7ffb34de8640 5 monclient: authenticate success, global_id 4340 2024-03-25T18:28:07.364 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.362+0000 7ffb34de8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:07.364 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.362+0000 7ffb2dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:07.364 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.362+0000 7ffb2dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:07.365 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.362+0000 7ffb2dd9b640 10 monclient: dump: 2024-03-25T18:28:07.365 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:07.365 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:07.365 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:07.365 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:07.365 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:07.365 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:07.365 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:07.365 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:07.365 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:07.365 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:07.365 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.362+0000 7ffb34de8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:07.367 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.366+0000 7ffb2dd9b640 1 -- 172.21.15.3:0/3299533495 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb180017f0 con 0x7ffb300b4670 2024-03-25T18:28:07.367 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.366+0000 7ffb2dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:07.367 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.366+0000 7ffb2dd9b640 1 -- 172.21.15.3:0/3299533495 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7ffb18001c00 con 0x7ffb300b4670 2024-03-25T18:28:07.367 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.366+0000 7ffb34de8640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:07.367 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.366+0000 7ffb34de8640 4 set_mon_vals no callback set 2024-03-25T18:28:07.367 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.366+0000 7ffb2dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:07.367 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.366+0000 7ffb2dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:07.367 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.366+0000 7ffb2dd9b640 10 monclient: dump: 2024-03-25T18:28:07.367 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:07.367 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:07.368 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:07.368 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:07.368 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:07.368 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:07.368 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:07.368 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:07.368 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:07.368 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:07.370 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.366+0000 7ffb34de8640 10 monclient: shutdown 2024-03-25T18:28:07.370 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.366+0000 7ffb34de8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:07.370 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.366+0000 7ffb34de8640 1 -- 172.21.15.3:0/3299533495 >> v1:172.21.15.3:6790/0 conn(0x7ffb300b4670 legacy=0x7ffb301425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:07.370 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.367+0000 7ffb34de8640 1 -- 172.21.15.3:0/3299533495 shutdown_connections 2024-03-25T18:28:07.371 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.367+0000 7ffb34de8640 1 -- 172.21.15.3:0/3299533495 >> 172.21.15.3:0/3299533495 conn(0x7ffb300617a0 msgr2=0x7ffb30061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:07.371 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.367+0000 7ffb34de8640 1 -- 172.21.15.3:0/3299533495 shutdown_connections 2024-03-25T18:28:07.371 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.367+0000 7ffb34de8640 1 -- 172.21.15.3:0/3299533495 wait complete. 2024-03-25T18:28:07.371 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.367+0000 7ffb34de8640 10 monclient: build_initial_monmap 2024-03-25T18:28:07.371 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.367+0000 7ffb34de8640 10 monclient: monmap: 2024-03-25T18:28:07.371 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:28:07.371 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:07.371 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:28:07.371 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:28:07.371 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:07.371 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:28:07.371 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:28:07.371 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:28:07.371 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:28:07.372 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:07.372 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.368+0000 7ffb34de8640 1 librados: starting msgr at 2024-03-25T18:28:07.372 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.368+0000 7ffb34de8640 1 librados: starting objecter 2024-03-25T18:28:07.372 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.368+0000 7ffb34de8640 1 Processor -- start 2024-03-25T18:28:07.372 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.368+0000 7ffb34de8640 1 -- start start 2024-03-25T18:28:07.372 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.368+0000 7ffb34de8640 1 librados: setting wanted keys 2024-03-25T18:28:07.372 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.368+0000 7ffb34de8640 1 librados: calling monclient init 2024-03-25T18:28:07.372 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.368+0000 7ffb34de8640 10 monclient: init 2024-03-25T18:28:07.372 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.369+0000 7ffb34de8640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:07.372 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.369+0000 7ffb34de8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:07.372 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.369+0000 7ffb34de8640 10 monclient(hunting): picked mon.noname-c con 0x7ffb300b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:28:07.372 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.369+0000 7ffb34de8640 10 monclient(hunting): picked mon.noname-b con 0x7ffb300b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:28:07.372 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.369+0000 7ffb34de8640 10 monclient(hunting): picked mon.noname-a con 0x7ffb3013c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:28:07.373 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.369+0000 7ffb34de8640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb3013afa0 con 0x7ffb3013c480 2024-03-25T18:28:07.373 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.369+0000 7ffb34de8640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb300afee0 con 0x7ffb300b4ff0 2024-03-25T18:28:07.373 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.369+0000 7ffb34de8640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb300b01c0 con 0x7ffb300b4670 2024-03-25T18:28:07.373 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.369+0000 7ffb34de8640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:07.373 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.369+0000 7ffb34de8640 10 monclient(hunting): authenticate will time out at 7541.629680s 2024-03-25T18:28:07.375 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.374+0000 7ffb27fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ffb300b4ff0 0x7ffb3013a180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:40976/0 (socket says 172.21.15.3:40976) 2024-03-25T18:28:07.375 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.374+0000 7ffb27fff640 1 -- 172.21.15.3:0/105542648 learned_addr learned my addr 172.21.15.3:0/105542648 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:28:07.375 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.374+0000 7ffb26ffd640 10 client.?.objecter ms_handle_connect 0x7ffb300b4ff0 2024-03-25T18:28:07.375 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.374+0000 7ffb26ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:07.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.383+0000 7ffb26ffd640 1 -- 172.21.15.3:0/105542648 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1974085009 0 0) 0x7ffb300afee0 con 0x7ffb300b4ff0 2024-03-25T18:28:07.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.383+0000 7ffb26ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:07.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.383+0000 7ffb26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:07.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.383+0000 7ffb26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:07.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.383+0000 7ffb26ffd640 10 monclient(hunting): my global_id is 4336 2024-03-25T18:28:07.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.383+0000 7ffb26ffd640 1 -- 172.21.15.3:0/105542648 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb10004eb0 con 0x7ffb300b4ff0 2024-03-25T18:28:07.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.386+0000 7ffb26ffd640 10 client.?.objecter ms_handle_connect 0x7ffb300b4670 2024-03-25T18:28:07.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.386+0000 7ffb26ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:07.394 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.393+0000 7ffb26ffd640 1 -- 172.21.15.3:0/105542648 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4158334842 0 0) 0x7ffb300b01c0 con 0x7ffb300b4670 2024-03-25T18:28:07.394 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.393+0000 7ffb26ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:07.394 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.393+0000 7ffb26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:07.394 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.393+0000 7ffb26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:07.394 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.393+0000 7ffb26ffd640 10 monclient(hunting): my global_id is 4343 2024-03-25T18:28:07.394 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.393+0000 7ffb26ffd640 1 -- 172.21.15.3:0/105542648 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb300afee0 con 0x7ffb300b4670 2024-03-25T18:28:07.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.393+0000 7ffb26ffd640 1 -- 172.21.15.3:0/105542648 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4289000139 0 0) 0x7ffb10004eb0 con 0x7ffb300b4ff0 2024-03-25T18:28:07.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.394+0000 7ffb26ffd640 1 -- 172.21.15.3:0/105542648 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb300b01c0 con 0x7ffb300b4ff0 2024-03-25T18:28:07.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.394+0000 7ffb26ffd640 1 -- 172.21.15.3:0/105542648 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7ffb18001bc0 con 0x7ffb300b4ff0 2024-03-25T18:28:07.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.394+0000 7ffb26ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:07.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.394+0000 7ffb26ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:07.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.394+0000 7ffb26ffd640 10 monclient(hunting): dump: 2024-03-25T18:28:07.396 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:07.396 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:07.396 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:07.396 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:07.396 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:07.396 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:07.396 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:07.396 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:07.396 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:07.396 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:07.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.394+0000 7ffb26ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:07.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.396+0000 7ffb26ffd640 10 client.?.objecter ms_handle_connect 0x7ffb3013c480 2024-03-25T18:28:07.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.396+0000 7ffb26ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:07.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.406+0000 7ffb26ffd640 1 -- 172.21.15.3:0/105542648 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 705874894 0 0) 0x7ffb300afee0 con 0x7ffb300b4670 2024-03-25T18:28:07.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.406+0000 7ffb26ffd640 1 -- 172.21.15.3:0/105542648 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb10004eb0 con 0x7ffb300b4670 2024-03-25T18:28:07.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.406+0000 7ffb26ffd640 1 -- 172.21.15.3:0/105542648 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7ffb20001970 con 0x7ffb300b4670 2024-03-25T18:28:07.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.406+0000 7ffb26ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:07.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.406+0000 7ffb26ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:07.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.406+0000 7ffb26ffd640 10 monclient(hunting): dump: 2024-03-25T18:28:07.407 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:07.407 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:07.408 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:07.408 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:07.408 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:07.408 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:07.408 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:07.408 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:07.408 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:07.408 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:07.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.410+0000 7ffb26ffd640 1 -- 172.21.15.3:0/105542648 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 564105702 0 0) 0x7ffb300b01c0 con 0x7ffb300b4ff0 2024-03-25T18:28:07.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.410+0000 7ffb26ffd640 1 -- 172.21.15.3:0/105542648 >> v1:172.21.15.3:6790/0 conn(0x7ffb300b4670 legacy=0x7ffb300b0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:07.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.410+0000 7ffb26ffd640 1 -- 172.21.15.3:0/105542648 >> v1:172.21.15.3:6789/0 conn(0x7ffb3013c480 legacy=0x7ffb3013a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:07.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.410+0000 7ffb26ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:28:07.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.410+0000 7ffb26ffd640 1 monclient: found mon.b 2024-03-25T18:28:07.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.410+0000 7ffb26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:07.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.410+0000 7ffb26ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:28:07.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.410+0000 7ffb26ffd640 1 -- 172.21.15.3:0/105542648 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb300b04a0 con 0x7ffb300b4ff0 2024-03-25T18:28:07.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.410+0000 7ffb26ffd640 10 monclient: _finish_auth 0 2024-03-25T18:28:07.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.410+0000 7ffb26ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:28:07.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.410+0000 7ffb26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:07.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.410+0000 7ffb34de8640 5 monclient: authenticate success, global_id 4336 2024-03-25T18:28:07.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.410+0000 7ffb34de8640 10 monclient: _renew_subs 2024-03-25T18:28:07.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.410+0000 7ffb34de8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:28:07.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.410+0000 7ffb34de8640 1 -- 172.21.15.3:0/105542648 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb300b0780 con 0x7ffb300b4ff0 2024-03-25T18:28:07.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.410+0000 7ffb34de8640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:07.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.410+0000 7ffb34de8640 10 monclient: _renew_subs 2024-03-25T18:28:07.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.410+0000 7ffb34de8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:28:07.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.410+0000 7ffb34de8640 1 -- 172.21.15.3:0/105542648 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb300b12b0 con 0x7ffb300b4ff0 2024-03-25T18:28:07.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.410+0000 7ffb34de8640 1 librados: init done 2024-03-25T18:28:07.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.411+0000 7ffb34de8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:07.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.411+0000 7ffb34de8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:07.414 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.411+0000 7ffb34de8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:28:07.414 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.411+0000 7ffb34de8640 1 -- 172.21.15.3:0/105542648 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb3006dbd0 con 0x7ffb300b4ff0 2024-03-25T18:28:07.706 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.705+0000 7ffb26ffd640 1 -- 172.21.15.3:0/105542648 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb180011f0 con 0x7ffb300b4ff0 2024-03-25T18:28:07.706 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.705+0000 7ffb26ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:07.706 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.705+0000 7ffb26ffd640 1 -- 172.21.15.3:0/105542648 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7ffb18003b00 con 0x7ffb300b4ff0 2024-03-25T18:28:07.706 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.705+0000 7ffb26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:07.707 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.705+0000 7ffb26ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:07.707 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.705+0000 7ffb26ffd640 10 monclient: dump: 2024-03-25T18:28:07.707 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:07.707 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:07.707 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:07.707 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:07.707 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:07.707 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:07.707 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:07.707 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:07.707 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:07.707 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:07.707 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.705+0000 7ffb2d59a640 4 set_mon_vals no callback set 2024-03-25T18:28:07.708 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.706+0000 7ffb26ffd640 1 -- 172.21.15.3:0/105542648 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7ffb18003d20 con 0x7ffb300b4ff0 2024-03-25T18:28:07.708 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.706+0000 7ffb2e59c640 10 monclient: get_auth_request con 0x7ffb1006ded0 auth_method 0 2024-03-25T18:28:07.708 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.707+0000 7ffb26ffd640 10 client.4336.objecter ms_handle_connect 0x7ffb1006ded0 2024-03-25T18:28:07.713 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.711+0000 7ffb26ffd640 1 -- 172.21.15.3:0/105542648 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7ffb180548d0 con 0x7ffb300b4ff0 2024-03-25T18:28:07.713 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.711+0000 7ffb26ffd640 10 client.4336.objecter ms_dispatch 0x7ffb30000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:07.713 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.711+0000 7ffb26ffd640 3 client.4336.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:07.713 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.711+0000 7ffb26ffd640 3 client.4336.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:07.713 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.712+0000 7ffb26ffd640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-25T18:28:07.713 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.712+0000 7ffb26ffd640 1 -- 172.21.15.3:0/105542648 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffb180503e0 con 0x7ffb300b4ff0 2024-03-25T18:28:07.713 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.712+0000 7ffb26ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:07.713 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.712+0000 7ffb26ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:07.976 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.975+0000 7ffb34de8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:07.976 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.975+0000 7ffb34de8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:07.976 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.975+0000 7ffb34de8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:28:07.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:07.975+0000 7ffb34de8640 1 -- 172.21.15.3:0/105542648 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb300688c0 con 0x7ffb300b4ff0 2024-03-25T18:28:08.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:08.101+0000 7ffb26ffd640 1 -- 172.21.15.3:0/105542648 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7ffb18052310 con 0x7ffb300b4ff0 2024-03-25T18:28:08.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:08.101+0000 7ffb26ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:08.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:08.101+0000 7ffb26ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:08.103 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:28:08.103 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:08.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:08.103+0000 7ffb34de8640 10 librados: watch_flush enter 2024-03-25T18:28:08.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:08.103+0000 7ffb34de8640 10 librados: watch_flush exit 2024-03-25T18:28:08.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:08.103+0000 7ffb34de8640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-25T18:28:08.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:08.103+0000 7ffb34de8640 10 client.4336.objecter successfully canceled tick 2024-03-25T18:28:08.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:08.103+0000 7ffb34de8640 1 -- 172.21.15.3:0/105542648 >> v1:172.21.15.3:6800/4195140966 conn(0x7ffb1006ded0 legacy=0x7ffb10070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:08.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:08.103+0000 7ffb34de8640 10 monclient: shutdown 2024-03-25T18:28:08.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:08.103+0000 7ffb34de8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:08.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:08.103+0000 7ffb34de8640 1 -- 172.21.15.3:0/105542648 >> v1:172.21.15.16:6789/0 conn(0x7ffb300b4ff0 legacy=0x7ffb3013a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:08.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:08.103+0000 7ffb34de8640 1 -- 172.21.15.3:0/105542648 shutdown_connections 2024-03-25T18:28:08.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:08.103+0000 7ffb34de8640 1 -- 172.21.15.3:0/105542648 >> 172.21.15.3:0/105542648 conn(0x7ffb300617a0 msgr2=0x7ffb30140740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:08.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:08.103+0000 7ffb34de8640 1 -- 172.21.15.3:0/105542648 shutdown_connections 2024-03-25T18:28:08.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:08.103+0000 7ffb34de8640 1 -- 172.21.15.3:0/105542648 wait complete. 2024-03-25T18:28:08.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:08.103+0000 7ffb34de8640 1 librados: shutdown 2024-03-25T18:28:08.116 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:14.923 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:15.038 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.037+0000 7efe1f19d640 10 monclient: get_monmap_and_config 2024-03-25T18:28:15.038 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.037+0000 7efe1f19d640 10 monclient: build_initial_monmap 2024-03-25T18:28:15.038 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.037+0000 7efe1f19d640 10 monclient: monmap: 2024-03-25T18:28:15.038 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:28:15.038 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:15.038 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:28:15.038345+0000 2024-03-25T18:28:15.039 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:28:15.038345+0000 2024-03-25T18:28:15.039 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:15.039 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:15.039 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:28:15.039 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:28:15.039 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:28:15.039 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:15.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.038+0000 7efe1f19d640 1 Processor -- start 2024-03-25T18:28:15.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.038+0000 7efe1f19d640 1 -- start start 2024-03-25T18:28:15.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.038+0000 7efe1f19d640 10 monclient: init 2024-03-25T18:28:15.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.038+0000 7efe1f19d640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:15.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.038+0000 7efe1f19d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:15.041 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.038+0000 7efe1f19d640 10 monclient(hunting): picked mon.noname-b con 0x7efe180615c0 addr v1:172.21.15.3:6790/0 2024-03-25T18:28:15.041 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.038+0000 7efe1f19d640 10 monclient(hunting): picked mon.noname-c con 0x7efe180619a0 addr v1:172.21.15.16:6789/0 2024-03-25T18:28:15.041 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.038+0000 7efe1f19d640 10 monclient(hunting): picked mon.noname-a con 0x7efe180aff00 addr v1:172.21.15.3:6789/0 2024-03-25T18:28:15.041 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.038+0000 7efe1f19d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe1806ef20 con 0x7efe180aff00 2024-03-25T18:28:15.041 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.038+0000 7efe1f19d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe1806f200 con 0x7efe180619a0 2024-03-25T18:28:15.041 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.038+0000 7efe1f19d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe1806f4e0 con 0x7efe180615c0 2024-03-25T18:28:15.041 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.038+0000 7efe1f19d640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:15.041 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.038+0000 7efe1f19d640 10 monclient(hunting): authenticate will time out at 7549.299189s 2024-03-25T18:28:15.049 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.048+0000 7efe17fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7efe180619a0 0x7efe18145890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:59414/0 (socket says 172.21.15.3:59414) 2024-03-25T18:28:15.049 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.048+0000 7efe17fff640 1 -- 172.21.15.3:0/408103919 learned_addr learned my addr 172.21.15.3:0/408103919 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:28:15.058 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.056+0000 7efe177fe640 1 -- 172.21.15.3:0/408103919 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2497123857 0 0) 0x7efe1806f200 con 0x7efe180619a0 2024-03-25T18:28:15.058 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.057+0000 7efe177fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:15.058 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.057+0000 7efe177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:15.058 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.057+0000 7efe177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:15.058 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.057+0000 7efe177fe640 10 monclient(hunting): my global_id is 4339 2024-03-25T18:28:15.058 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.057+0000 7efe177fe640 1 -- 172.21.15.3:0/408103919 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe08003520 con 0x7efe180619a0 2024-03-25T18:28:15.073 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.072+0000 7efe177fe640 1 -- 172.21.15.3:0/408103919 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2997377658 0 0) 0x7efe1806f4e0 con 0x7efe180615c0 2024-03-25T18:28:15.073 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.072+0000 7efe177fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:15.073 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.072+0000 7efe177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:15.073 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.072+0000 7efe177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:15.073 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.072+0000 7efe177fe640 10 monclient(hunting): my global_id is 4346 2024-03-25T18:28:15.073 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.072+0000 7efe177fe640 1 -- 172.21.15.3:0/408103919 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe1806f200 con 0x7efe180615c0 2024-03-25T18:28:15.074 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.073+0000 7efe177fe640 1 -- 172.21.15.3:0/408103919 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4216756875 0 0) 0x7efe1806ef20 con 0x7efe180aff00 2024-03-25T18:28:15.074 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.073+0000 7efe177fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:15.074 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.073+0000 7efe177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:15.074 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.073+0000 7efe177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:15.074 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.073+0000 7efe177fe640 10 monclient(hunting): my global_id is 4377 2024-03-25T18:28:15.074 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.073+0000 7efe177fe640 1 -- 172.21.15.3:0/408103919 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe1806f4e0 con 0x7efe180aff00 2024-03-25T18:28:15.075 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.074+0000 7efe177fe640 1 -- 172.21.15.3:0/408103919 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1045848440 0 0) 0x7efe08003520 con 0x7efe180619a0 2024-03-25T18:28:15.075 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.074+0000 7efe177fe640 1 -- 172.21.15.3:0/408103919 >> v1:172.21.15.3:6790/0 conn(0x7efe180615c0 legacy=0x7efe18071f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:15.076 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.074+0000 7efe177fe640 1 -- 172.21.15.3:0/408103919 >> v1:172.21.15.3:6789/0 conn(0x7efe180aff00 legacy=0x7efe1814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:15.076 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.074+0000 7efe177fe640 10 monclient: _finish_hunting 0 2024-03-25T18:28:15.076 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.074+0000 7efe177fe640 1 monclient: found mon.noname-c 2024-03-25T18:28:15.076 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.074+0000 7efe177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:15.076 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.074+0000 7efe177fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:28:15.076 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.074+0000 7efe177fe640 1 -- 172.21.15.3:0/408103919 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe1813fe50 con 0x7efe180619a0 2024-03-25T18:28:15.076 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.074+0000 7efe177fe640 10 monclient: _finish_auth 0 2024-03-25T18:28:15.076 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.074+0000 7efe177fe640 10 monclient: _check_auth_tickets 2024-03-25T18:28:15.076 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.074+0000 7efe177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:15.076 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.074+0000 7efe177fe640 1 -- 172.21.15.3:0/408103919 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7efe04001690 con 0x7efe180619a0 2024-03-25T18:28:15.076 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.074+0000 7efe177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:15.076 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.074+0000 7efe177fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:15.076 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.074+0000 7efe177fe640 10 monclient: dump: 2024-03-25T18:28:15.077 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:15.077 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:15.077 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:15.077 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:15.077 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:15.077 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:15.077 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:15.077 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:15.077 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:15.077 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:15.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.074+0000 7efe1f19d640 5 monclient: authenticate success, global_id 4339 2024-03-25T18:28:15.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.074+0000 7efe1f19d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:15.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.102+0000 7efe177fe640 1 -- 172.21.15.3:0/408103919 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe040017f0 con 0x7efe180619a0 2024-03-25T18:28:15.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.102+0000 7efe177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:15.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.102+0000 7efe177fe640 1 -- 172.21.15.3:0/408103919 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7efe04001c00 con 0x7efe180619a0 2024-03-25T18:28:15.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.102+0000 7efe1f19d640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:15.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.102+0000 7efe1f19d640 4 set_mon_vals no callback set 2024-03-25T18:28:15.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.102+0000 7efe177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:15.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.102+0000 7efe177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:15.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.102+0000 7efe177fe640 10 monclient: dump: 2024-03-25T18:28:15.104 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:15.104 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:15.104 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:15.104 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:15.105 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:15.105 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:15.105 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:15.105 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:15.105 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:15.105 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:15.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.103+0000 7efe1f19d640 10 monclient: shutdown 2024-03-25T18:28:15.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.103+0000 7efe1f19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:15.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.103+0000 7efe1f19d640 1 -- 172.21.15.3:0/408103919 >> v1:172.21.15.16:6789/0 conn(0x7efe180619a0 legacy=0x7efe18145890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:15.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.103+0000 7efe1f19d640 1 -- 172.21.15.3:0/408103919 shutdown_connections 2024-03-25T18:28:15.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.103+0000 7efe1f19d640 1 -- 172.21.15.3:0/408103919 >> 172.21.15.3:0/408103919 conn(0x7efe180b1020 msgr2=0x7efe180b1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:15.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.103+0000 7efe1f19d640 1 -- 172.21.15.3:0/408103919 shutdown_connections 2024-03-25T18:28:15.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.103+0000 7efe1f19d640 1 -- 172.21.15.3:0/408103919 wait complete. 2024-03-25T18:28:15.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.104+0000 7efe1f19d640 10 monclient: build_initial_monmap 2024-03-25T18:28:15.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.104+0000 7efe1f19d640 10 monclient: monmap: 2024-03-25T18:28:15.107 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:28:15.107 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:15.107 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:28:15.107 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:28:15.107 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:15.107 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:28:15.107 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:28:15.107 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:28:15.107 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:28:15.108 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:15.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.105+0000 7efe1f19d640 1 librados: starting msgr at 2024-03-25T18:28:15.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.105+0000 7efe1f19d640 1 librados: starting objecter 2024-03-25T18:28:15.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.105+0000 7efe1f19d640 1 Processor -- start 2024-03-25T18:28:15.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.105+0000 7efe1f19d640 1 -- start start 2024-03-25T18:28:15.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.105+0000 7efe1f19d640 1 librados: setting wanted keys 2024-03-25T18:28:15.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.105+0000 7efe1f19d640 1 librados: calling monclient init 2024-03-25T18:28:15.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.105+0000 7efe1f19d640 10 monclient: init 2024-03-25T18:28:15.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.106+0000 7efe1f19d640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:15.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.106+0000 7efe1f19d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:15.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.106+0000 7efe1f19d640 10 monclient(hunting): picked mon.noname-b con 0x7efe180615c0 addr v1:172.21.15.16:6789/0 2024-03-25T18:28:15.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.106+0000 7efe1f19d640 10 monclient(hunting): picked mon.noname-a con 0x7efe180aff00 addr v1:172.21.15.3:6789/0 2024-03-25T18:28:15.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.106+0000 7efe1f19d640 10 monclient(hunting): picked mon.noname-c con 0x7efe1813e3d0 addr v1:172.21.15.3:6790/0 2024-03-25T18:28:15.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.106+0000 7efe1f19d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe181d7660 con 0x7efe180aff00 2024-03-25T18:28:15.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.106+0000 7efe1f19d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe18137780 con 0x7efe180615c0 2024-03-25T18:28:15.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.106+0000 7efe1f19d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe181d84e0 con 0x7efe1813e3d0 2024-03-25T18:28:15.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.106+0000 7efe1f19d640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:15.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.106+0000 7efe1f19d640 10 monclient(hunting): authenticate will time out at 7549.366852s 2024-03-25T18:28:15.115 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.114+0000 7efe1cf39640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7efe180615c0 0x7efe181d7dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:59454/0 (socket says 172.21.15.3:59454) 2024-03-25T18:28:15.115 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.114+0000 7efe1cf39640 1 -- 172.21.15.3:0/1345562217 learned_addr learned my addr 172.21.15.3:0/1345562217 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:28:15.116 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.115+0000 7efe157fa640 10 client.?.objecter ms_handle_connect 0x7efe180615c0 2024-03-25T18:28:15.116 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.115+0000 7efe157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:15.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.122+0000 7efe157fa640 10 client.?.objecter ms_handle_connect 0x7efe1813e3d0 2024-03-25T18:28:15.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.122+0000 7efe157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:15.127 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.126+0000 7efe157fa640 1 -- 172.21.15.3:0/1345562217 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2658672349 0 0) 0x7efe18137780 con 0x7efe180615c0 2024-03-25T18:28:15.127 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.126+0000 7efe157fa640 10 monclient(hunting): _negotiate 2024-03-25T18:28:15.127 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.126+0000 7efe157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:15.127 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.126+0000 7efe157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:15.127 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.126+0000 7efe157fa640 10 monclient(hunting): my global_id is 4342 2024-03-25T18:28:15.127 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.126+0000 7efe157fa640 1 -- 172.21.15.3:0/1345562217 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdfc004eb0 con 0x7efe180615c0 2024-03-25T18:28:15.136 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.135+0000 7efe157fa640 10 client.?.objecter ms_handle_connect 0x7efe180aff00 2024-03-25T18:28:15.136 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.135+0000 7efe157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:15.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.139+0000 7efe157fa640 1 -- 172.21.15.3:0/1345562217 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 342512781 0 0) 0x7efe181d84e0 con 0x7efe1813e3d0 2024-03-25T18:28:15.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.139+0000 7efe157fa640 10 monclient(hunting): _negotiate 2024-03-25T18:28:15.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.139+0000 7efe157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:15.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.139+0000 7efe157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:15.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.139+0000 7efe157fa640 10 monclient(hunting): my global_id is 4349 2024-03-25T18:28:15.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.139+0000 7efe157fa640 1 -- 172.21.15.3:0/1345562217 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe18137780 con 0x7efe1813e3d0 2024-03-25T18:28:15.149 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.148+0000 7efe157fa640 1 -- 172.21.15.3:0/1345562217 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1362901503 0 0) 0x7efdfc004eb0 con 0x7efe180615c0 2024-03-25T18:28:15.149 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.148+0000 7efe157fa640 1 -- 172.21.15.3:0/1345562217 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe181d84e0 con 0x7efe180615c0 2024-03-25T18:28:15.150 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.148+0000 7efe157fa640 1 -- 172.21.15.3:0/1345562217 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7efe0c001990 con 0x7efe180615c0 2024-03-25T18:28:15.150 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.148+0000 7efe157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:15.150 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.148+0000 7efe157fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:15.150 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.148+0000 7efe157fa640 10 monclient(hunting): dump: 2024-03-25T18:28:15.150 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:15.150 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:15.150 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:15.150 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:15.150 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:15.150 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:15.150 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:15.150 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:15.151 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:15.151 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:15.151 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.148+0000 7efe157fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:15.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.154+0000 7efe157fa640 1 -- 172.21.15.3:0/1345562217 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3829916857 0 0) 0x7efe181d7660 con 0x7efe180aff00 2024-03-25T18:28:15.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.154+0000 7efe157fa640 10 monclient(hunting): _negotiate 2024-03-25T18:28:15.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.154+0000 7efe157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:15.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.154+0000 7efe157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:15.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.154+0000 7efe157fa640 10 monclient(hunting): my global_id is 4380 2024-03-25T18:28:15.156 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.154+0000 7efe157fa640 1 -- 172.21.15.3:0/1345562217 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdfc004eb0 con 0x7efe180aff00 2024-03-25T18:28:15.161 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.160+0000 7efe157fa640 1 -- 172.21.15.3:0/1345562217 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4012823852 0 0) 0x7efe18137780 con 0x7efe1813e3d0 2024-03-25T18:28:15.161 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.160+0000 7efe157fa640 1 -- 172.21.15.3:0/1345562217 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe181d7660 con 0x7efe1813e3d0 2024-03-25T18:28:15.161 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.160+0000 7efe157fa640 1 -- 172.21.15.3:0/1345562217 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7efe10001bf0 con 0x7efe1813e3d0 2024-03-25T18:28:15.161 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.160+0000 7efe157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:15.161 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.160+0000 7efe157fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:15.161 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.160+0000 7efe157fa640 10 monclient(hunting): dump: 2024-03-25T18:28:15.161 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:15.161 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:15.162 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:15.162 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:15.162 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:15.162 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:15.162 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:15.162 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:15.162 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:15.162 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:15.163 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.162+0000 7efe157fa640 1 -- 172.21.15.3:0/1345562217 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2922614634 0 0) 0x7efe181d84e0 con 0x7efe180615c0 2024-03-25T18:28:15.163 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.162+0000 7efe157fa640 1 -- 172.21.15.3:0/1345562217 >> v1:172.21.15.3:6790/0 conn(0x7efe1813e3d0 legacy=0x7efe181e2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:15.163 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.162+0000 7efe157fa640 1 -- 172.21.15.3:0/1345562217 >> v1:172.21.15.3:6789/0 conn(0x7efe180aff00 legacy=0x7efe1813dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:15.163 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.162+0000 7efe157fa640 10 monclient: _finish_hunting 0 2024-03-25T18:28:15.163 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.162+0000 7efe157fa640 1 monclient: found mon.b 2024-03-25T18:28:15.164 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.162+0000 7efe157fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:15.164 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.162+0000 7efe157fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:28:15.164 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.162+0000 7efe157fa640 1 -- 172.21.15.3:0/1345562217 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe181379f0 con 0x7efe180615c0 2024-03-25T18:28:15.164 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.162+0000 7efe157fa640 10 monclient: _finish_auth 0 2024-03-25T18:28:15.164 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.162+0000 7efe157fa640 10 monclient: _check_auth_tickets 2024-03-25T18:28:15.164 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.162+0000 7efe157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:15.165 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.162+0000 7efe1f19d640 5 monclient: authenticate success, global_id 4342 2024-03-25T18:28:15.165 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.162+0000 7efe1f19d640 10 monclient: _renew_subs 2024-03-25T18:28:15.165 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.162+0000 7efe1f19d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:28:15.165 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.162+0000 7efe1f19d640 1 -- 172.21.15.3:0/1345562217 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe181d78b0 con 0x7efe180615c0 2024-03-25T18:28:15.165 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.162+0000 7efe1f19d640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:15.165 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.162+0000 7efe1f19d640 10 monclient: _renew_subs 2024-03-25T18:28:15.165 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.162+0000 7efe1f19d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:28:15.165 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.162+0000 7efe1f19d640 1 -- 172.21.15.3:0/1345562217 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe181d42d0 con 0x7efe180615c0 2024-03-25T18:28:15.165 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.162+0000 7efe1f19d640 1 librados: init done 2024-03-25T18:28:15.165 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.163+0000 7efe1f19d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:15.165 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.163+0000 7efe1f19d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:15.165 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.163+0000 7efe1f19d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:28:15.166 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.163+0000 7efe1f19d640 1 -- 172.21.15.3:0/1345562217 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe1806bd80 con 0x7efe180615c0 2024-03-25T18:28:15.455 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.453+0000 7efe157fa640 1 -- 172.21.15.3:0/1345562217 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe0c0035f0 con 0x7efe180615c0 2024-03-25T18:28:15.455 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.453+0000 7efe157fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:15.455 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.454+0000 7efe157fa640 1 -- 172.21.15.3:0/1345562217 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7efe0c003a00 con 0x7efe180615c0 2024-03-25T18:28:15.455 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.454+0000 7efe157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:15.455 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.454+0000 7efe157fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:15.455 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.454+0000 7efe157fa640 10 monclient: dump: 2024-03-25T18:28:15.455 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:15.456 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:15.456 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:15.456 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:15.456 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:15.456 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:15.456 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:15.456 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:15.456 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:15.456 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:15.456 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.454+0000 7efe16ffd640 4 set_mon_vals no callback set 2024-03-25T18:28:15.457 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.454+0000 7efe157fa640 1 -- 172.21.15.3:0/1345562217 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7efe0c003c20 con 0x7efe180615c0 2024-03-25T18:28:15.457 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.455+0000 7efe17fff640 10 monclient: get_auth_request con 0x7efdfc06dfb0 auth_method 0 2024-03-25T18:28:15.457 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.455+0000 7efe157fa640 10 client.4342.objecter ms_handle_connect 0x7efdfc06dfb0 2024-03-25T18:28:15.462 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.460+0000 7efe157fa640 1 -- 172.21.15.3:0/1345562217 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7efe0c0547d0 con 0x7efe180615c0 2024-03-25T18:28:15.462 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.461+0000 7efe157fa640 10 client.4342.objecter ms_dispatch 0x7efe18000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:15.462 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.461+0000 7efe157fa640 3 client.4342.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:15.462 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.461+0000 7efe157fa640 3 client.4342.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:15.462 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.461+0000 7efe157fa640 20 client.4342.objecter dump_active .. 0 homeless 2024-03-25T18:28:15.462 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.461+0000 7efe157fa640 1 -- 172.21.15.3:0/1345562217 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efe0c0502e0 con 0x7efe180615c0 2024-03-25T18:28:15.462 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.461+0000 7efe157fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:15.463 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.461+0000 7efe157fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:15.711 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.710+0000 7efe1f19d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:15.711 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.710+0000 7efe1f19d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:15.711 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.710+0000 7efe1f19d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:28:15.711 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.710+0000 7efe1f19d640 1 -- 172.21.15.3:0/1345562217 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe181d7a40 con 0x7efe180615c0 2024-03-25T18:28:15.826 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.824+0000 7efe157fa640 1 -- 172.21.15.3:0/1345562217 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7efe0c052210 con 0x7efe180615c0 2024-03-25T18:28:15.826 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.824+0000 7efe157fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:15.826 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.824+0000 7efe157fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:15.827 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:28:15.827 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:15.829 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.827+0000 7efe1f19d640 10 librados: watch_flush enter 2024-03-25T18:28:15.829 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.827+0000 7efe1f19d640 10 librados: watch_flush exit 2024-03-25T18:28:15.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.827+0000 7efe1f19d640 20 client.4342.objecter shutdown clearing up homeless session... 2024-03-25T18:28:15.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.827+0000 7efe1f19d640 10 client.4342.objecter successfully canceled tick 2024-03-25T18:28:15.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.827+0000 7efe1f19d640 1 -- 172.21.15.3:0/1345562217 >> v1:172.21.15.3:6800/4195140966 conn(0x7efdfc06dfb0 legacy=0x7efdfc070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:15.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.827+0000 7efe1f19d640 10 monclient: shutdown 2024-03-25T18:28:15.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.827+0000 7efe1f19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:15.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.827+0000 7efe1f19d640 1 -- 172.21.15.3:0/1345562217 >> v1:172.21.15.16:6789/0 conn(0x7efe180615c0 legacy=0x7efe181d7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:15.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.827+0000 7efe1f19d640 1 -- 172.21.15.3:0/1345562217 shutdown_connections 2024-03-25T18:28:15.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.827+0000 7efe1f19d640 1 -- 172.21.15.3:0/1345562217 >> 172.21.15.3:0/1345562217 conn(0x7efe180b1020 msgr2=0x7efe1806f9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:15.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.827+0000 7efe1f19d640 1 -- 172.21.15.3:0/1345562217 shutdown_connections 2024-03-25T18:28:15.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.827+0000 7efe1f19d640 1 -- 172.21.15.3:0/1345562217 wait complete. 2024-03-25T18:28:15.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:15.827+0000 7efe1f19d640 1 librados: shutdown 2024-03-25T18:28:15.840 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:21.240 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:03:34.969 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:21.292 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:03:35.021 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:21.338 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:03:35.067 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:21.346 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:03:35.038 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:21.359 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:03:35.088 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:21.363 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:03:35.055 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:21.372 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:03:35.101 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:21.384 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:03:35.076 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:21.405 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:03:35.134 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:22.647 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:22.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.760+0000 7f8e6ea33640 10 monclient: get_monmap_and_config 2024-03-25T18:28:22.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.760+0000 7f8e6ea33640 10 monclient: build_initial_monmap 2024-03-25T18:28:22.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.760+0000 7f8e6ea33640 10 monclient: monmap: 2024-03-25T18:28:22.760 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:28:22.760 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:22.760 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:28:22.760202+0000 2024-03-25T18:28:22.760 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:28:22.760202+0000 2024-03-25T18:28:22.761 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:22.761 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:22.761 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:28:22.761 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:28:22.761 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:28:22.761 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:22.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.761+0000 7f8e6ea33640 1 Processor -- start 2024-03-25T18:28:22.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.761+0000 7f8e6ea33640 1 -- start start 2024-03-25T18:28:22.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.761+0000 7f8e6ea33640 10 monclient: init 2024-03-25T18:28:22.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.761+0000 7f8e6ea33640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:22.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.761+0000 7f8e6ea33640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:22.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.761+0000 7f8e6ea33640 10 monclient(hunting): picked mon.noname-b con 0x7f8e680b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:28:22.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.761+0000 7f8e6ea33640 10 monclient(hunting): picked mon.noname-c con 0x7f8e680b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:28:22.763 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.761+0000 7f8e6ea33640 10 monclient(hunting): picked mon.noname-a con 0x7f8e6813c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:28:22.763 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.761+0000 7f8e6ea33640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e680ab0e0 con 0x7f8e6813c480 2024-03-25T18:28:22.763 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.761+0000 7f8e6ea33640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e680ab3c0 con 0x7f8e680b4670 2024-03-25T18:28:22.763 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.761+0000 7f8e6ea33640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e680a9f30 con 0x7f8e680b4ff0 2024-03-25T18:28:22.763 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.761+0000 7f8e6ea33640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:22.763 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.761+0000 7f8e6ea33640 10 monclient(hunting): authenticate will time out at 7557.021167s 2024-03-25T18:28:22.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.765+0000 7f8e6cfd0640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f8e6813c480 0x7f8e6814dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54290/0 (socket says 172.21.15.3:54290) 2024-03-25T18:28:22.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.765+0000 7f8e6cfd0640 1 -- 172.21.15.3:0/3384041431 learned_addr learned my addr 172.21.15.3:0/3384041431 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:28:22.781 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.781+0000 7f8e66ffd640 1 -- 172.21.15.3:0/3384041431 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1819004204 0 0) 0x7f8e680ab0e0 con 0x7f8e6813c480 2024-03-25T18:28:22.781 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.781+0000 7f8e66ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:22.781 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.781+0000 7f8e66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:22.781 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.781+0000 7f8e66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:22.781 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.781+0000 7f8e66ffd640 10 monclient(hunting): my global_id is 4383 2024-03-25T18:28:22.781 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.781+0000 7f8e66ffd640 1 -- 172.21.15.3:0/3384041431 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e50003520 con 0x7f8e6813c480 2024-03-25T18:28:22.790 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.790+0000 7f8e66ffd640 1 -- 172.21.15.3:0/3384041431 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197112552 0 0) 0x7f8e680a9f30 con 0x7f8e680b4ff0 2024-03-25T18:28:22.790 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.790+0000 7f8e66ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:22.790 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.790+0000 7f8e66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:22.790 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.790+0000 7f8e66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:22.790 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.790+0000 7f8e66ffd640 10 monclient(hunting): my global_id is 4352 2024-03-25T18:28:22.790 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.790+0000 7f8e66ffd640 1 -- 172.21.15.3:0/3384041431 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e680ab0e0 con 0x7f8e680b4ff0 2024-03-25T18:28:22.795 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.795+0000 7f8e66ffd640 1 -- 172.21.15.3:0/3384041431 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4272970612 0 0) 0x7f8e50003520 con 0x7f8e6813c480 2024-03-25T18:28:22.795 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.795+0000 7f8e66ffd640 1 -- 172.21.15.3:0/3384041431 >> v1:172.21.15.3:6790/0 conn(0x7f8e680b4ff0 legacy=0x7f8e6813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:22.795 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.795+0000 7f8e66ffd640 1 -- 172.21.15.3:0/3384041431 >> v1:172.21.15.16:6789/0 conn(0x7f8e680b4670 legacy=0x7f8e681425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:22.795 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.795+0000 7f8e66ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:28:22.795 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.795+0000 7f8e66ffd640 1 monclient: found mon.noname-a 2024-03-25T18:28:22.796 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.795+0000 7f8e66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:22.796 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.795+0000 7f8e66ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:28:22.796 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.795+0000 7f8e66ffd640 1 -- 172.21.15.3:0/3384041431 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e680a9b70 con 0x7f8e6813c480 2024-03-25T18:28:22.796 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.795+0000 7f8e66ffd640 10 monclient: _finish_auth 0 2024-03-25T18:28:22.796 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.795+0000 7f8e66ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:28:22.796 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.795+0000 7f8e66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:22.796 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.795+0000 7f8e66ffd640 1 -- 172.21.15.3:0/3384041431 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f8e60001a10 con 0x7f8e6813c480 2024-03-25T18:28:22.796 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.795+0000 7f8e66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:22.796 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.795+0000 7f8e66ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:22.796 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.795+0000 7f8e66ffd640 10 monclient: dump: 2024-03-25T18:28:22.796 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:22.796 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:22.796 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:22.797 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:22.797 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:22.797 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:22.797 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:22.797 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:22.797 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:22.797 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:22.797 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.795+0000 7f8e6ea33640 5 monclient: authenticate success, global_id 4383 2024-03-25T18:28:22.797 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.795+0000 7f8e6ea33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:22.804 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.804+0000 7f8e66ffd640 1 -- 172.21.15.3:0/3384041431 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e60001b70 con 0x7f8e6813c480 2024-03-25T18:28:22.804 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.804+0000 7f8e66ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:22.804 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.804+0000 7f8e66ffd640 1 -- 172.21.15.3:0/3384041431 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f8e60001f80 con 0x7f8e6813c480 2024-03-25T18:28:22.804 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.804+0000 7f8e66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:22.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.804+0000 7f8e66ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:22.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.804+0000 7f8e66ffd640 10 monclient: dump: 2024-03-25T18:28:22.805 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:22.805 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:22.805 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:22.805 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:22.805 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:22.805 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:22.805 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:22.805 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:22.805 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:22.805 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:22.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.804+0000 7f8e6ea33640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:22.806 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.804+0000 7f8e6ea33640 4 set_mon_vals no callback set 2024-03-25T18:28:22.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.805+0000 7f8e6ea33640 10 monclient: shutdown 2024-03-25T18:28:22.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.805+0000 7f8e6ea33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:22.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.805+0000 7f8e6ea33640 1 -- 172.21.15.3:0/3384041431 >> v1:172.21.15.3:6789/0 conn(0x7f8e6813c480 legacy=0x7f8e6814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:22.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.805+0000 7f8e6ea33640 1 -- 172.21.15.3:0/3384041431 shutdown_connections 2024-03-25T18:28:22.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.805+0000 7f8e6ea33640 1 -- 172.21.15.3:0/3384041431 >> 172.21.15.3:0/3384041431 conn(0x7f8e680617a0 msgr2=0x7f8e68061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:22.808 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.805+0000 7f8e6ea33640 1 -- 172.21.15.3:0/3384041431 shutdown_connections 2024-03-25T18:28:22.808 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.805+0000 7f8e6ea33640 1 -- 172.21.15.3:0/3384041431 wait complete. 2024-03-25T18:28:22.809 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.806+0000 7f8e6ea33640 10 monclient: build_initial_monmap 2024-03-25T18:28:22.809 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.806+0000 7f8e6ea33640 10 monclient: monmap: 2024-03-25T18:28:22.809 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:28:22.809 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:22.809 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:28:22.809 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:28:22.809 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:22.810 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:28:22.810 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:28:22.810 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:28:22.810 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:28:22.810 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:22.810 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.806+0000 7f8e6ea33640 1 librados: starting msgr at 2024-03-25T18:28:22.810 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.806+0000 7f8e6ea33640 1 librados: starting objecter 2024-03-25T18:28:22.810 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.807+0000 7f8e6ea33640 1 Processor -- start 2024-03-25T18:28:22.810 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.807+0000 7f8e6ea33640 1 -- start start 2024-03-25T18:28:22.810 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.807+0000 7f8e6ea33640 1 librados: setting wanted keys 2024-03-25T18:28:22.810 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.807+0000 7f8e6ea33640 1 librados: calling monclient init 2024-03-25T18:28:22.810 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.807+0000 7f8e6ea33640 10 monclient: init 2024-03-25T18:28:22.811 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.808+0000 7f8e6ea33640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:22.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.808+0000 7f8e6ea33640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:22.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.808+0000 7f8e6ea33640 10 monclient(hunting): picked mon.noname-b con 0x7f8e680b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:28:22.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.808+0000 7f8e6ea33640 10 monclient(hunting): picked mon.noname-a con 0x7f8e680b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:28:22.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.808+0000 7f8e6ea33640 10 monclient(hunting): picked mon.noname-c con 0x7f8e6813c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:28:22.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.808+0000 7f8e6ea33640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e681465d0 con 0x7f8e680b4ff0 2024-03-25T18:28:22.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.808+0000 7f8e6ea33640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e68145660 con 0x7f8e680b4670 2024-03-25T18:28:22.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.808+0000 7f8e6ea33640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e681dbf50 con 0x7f8e6813c480 2024-03-25T18:28:22.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.808+0000 7f8e6ea33640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:22.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.808+0000 7f8e6ea33640 10 monclient(hunting): authenticate will time out at 7557.067765s 2024-03-25T18:28:22.817 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.817+0000 7f8e67fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8e680b4670 0x7f8e68144f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:57398/0 (socket says 172.21.15.3:57398) 2024-03-25T18:28:22.817 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.817+0000 7f8e67fff640 1 -- 172.21.15.3:0/1079881588 learned_addr learned my addr 172.21.15.3:0/1079881588 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:28:22.817 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.817+0000 7f8e64ff9640 10 client.?.objecter ms_handle_connect 0x7f8e680b4670 2024-03-25T18:28:22.817 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.817+0000 7f8e64ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:22.818 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.818+0000 7f8e64ff9640 10 client.?.objecter ms_handle_connect 0x7f8e6813c480 2024-03-25T18:28:22.818 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.818+0000 7f8e64ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:22.824 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.823+0000 7f8e64ff9640 1 -- 172.21.15.3:0/1079881588 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1290438429 0 0) 0x7f8e681dbf50 con 0x7f8e6813c480 2024-03-25T18:28:22.824 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.823+0000 7f8e64ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:22.824 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.823+0000 7f8e64ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:22.824 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.823+0000 7f8e64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:22.824 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.823+0000 7f8e64ff9640 10 monclient(hunting): my global_id is 4355 2024-03-25T18:28:22.824 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.823+0000 7f8e64ff9640 1 -- 172.21.15.3:0/1079881588 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e54004eb0 con 0x7f8e6813c480 2024-03-25T18:28:22.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.831+0000 7f8e64ff9640 10 client.?.objecter ms_handle_connect 0x7f8e680b4ff0 2024-03-25T18:28:22.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.831+0000 7f8e64ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:22.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.831+0000 7f8e64ff9640 1 -- 172.21.15.3:0/1079881588 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1289896026 0 0) 0x7f8e68145660 con 0x7f8e680b4670 2024-03-25T18:28:22.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.831+0000 7f8e64ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:22.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.831+0000 7f8e64ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:22.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.831+0000 7f8e64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:22.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.831+0000 7f8e64ff9640 10 monclient(hunting): my global_id is 4348 2024-03-25T18:28:22.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.831+0000 7f8e64ff9640 1 -- 172.21.15.3:0/1079881588 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e681dbf50 con 0x7f8e680b4670 2024-03-25T18:28:22.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.834+0000 7f8e64ff9640 1 -- 172.21.15.3:0/1079881588 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1265677287 0 0) 0x7f8e54004eb0 con 0x7f8e6813c480 2024-03-25T18:28:22.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.834+0000 7f8e64ff9640 1 -- 172.21.15.3:0/1079881588 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e68145660 con 0x7f8e6813c480 2024-03-25T18:28:22.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.834+0000 7f8e64ff9640 1 -- 172.21.15.3:0/1079881588 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f8e60001ec0 con 0x7f8e6813c480 2024-03-25T18:28:22.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.834+0000 7f8e64ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:22.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.834+0000 7f8e64ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:22.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.834+0000 7f8e64ff9640 10 monclient(hunting): dump: 2024-03-25T18:28:22.835 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:22.835 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:22.835 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:22.835 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:22.835 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:22.835 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:22.835 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:22.836 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:22.836 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:22.836 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:22.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.834+0000 7f8e64ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:22.852 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.852+0000 7f8e64ff9640 1 -- 172.21.15.3:0/1079881588 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4090981920 0 0) 0x7f8e681465d0 con 0x7f8e680b4ff0 2024-03-25T18:28:22.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.852+0000 7f8e64ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:22.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.852+0000 7f8e64ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:22.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.852+0000 7f8e64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:22.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.852+0000 7f8e64ff9640 10 monclient(hunting): my global_id is 4386 2024-03-25T18:28:22.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.852+0000 7f8e64ff9640 1 -- 172.21.15.3:0/1079881588 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e54004eb0 con 0x7f8e680b4ff0 2024-03-25T18:28:22.855 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.854+0000 7f8e64ff9640 1 -- 172.21.15.3:0/1079881588 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2973721990 0 0) 0x7f8e68145660 con 0x7f8e6813c480 2024-03-25T18:28:22.855 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.855+0000 7f8e64ff9640 1 -- 172.21.15.3:0/1079881588 >> v1:172.21.15.16:6789/0 conn(0x7f8e680b4670 legacy=0x7f8e68144f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:22.855 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.855+0000 7f8e64ff9640 1 -- 172.21.15.3:0/1079881588 >> v1:172.21.15.3:6789/0 conn(0x7f8e680b4ff0 legacy=0x7f8e681db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:22.855 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.855+0000 7f8e64ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:28:22.855 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.855+0000 7f8e64ff9640 1 monclient: found mon.c 2024-03-25T18:28:22.855 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.855+0000 7f8e64ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:22.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.855+0000 7f8e64ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:28:22.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.855+0000 7f8e64ff9640 1 -- 172.21.15.3:0/1079881588 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e681444c0 con 0x7f8e6813c480 2024-03-25T18:28:22.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.855+0000 7f8e64ff9640 10 monclient: _finish_auth 0 2024-03-25T18:28:22.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.855+0000 7f8e64ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:28:22.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.855+0000 7f8e64ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:22.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.855+0000 7f8e6ea33640 5 monclient: authenticate success, global_id 4355 2024-03-25T18:28:22.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.855+0000 7f8e6ea33640 10 monclient: _renew_subs 2024-03-25T18:28:22.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.855+0000 7f8e6ea33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:28:22.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.855+0000 7f8e6ea33640 1 -- 172.21.15.3:0/1079881588 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e681447a0 con 0x7f8e6813c480 2024-03-25T18:28:22.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.855+0000 7f8e6ea33640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:22.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.855+0000 7f8e6ea33640 10 monclient: _renew_subs 2024-03-25T18:28:22.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.855+0000 7f8e6ea33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:28:22.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.855+0000 7f8e6ea33640 1 -- 172.21.15.3:0/1079881588 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e68143800 con 0x7f8e6813c480 2024-03-25T18:28:22.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.855+0000 7f8e6ea33640 1 librados: init done 2024-03-25T18:28:22.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.856+0000 7f8e6ea33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:22.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.856+0000 7f8e6ea33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:22.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.856+0000 7f8e6ea33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:28:22.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:22.856+0000 7f8e6ea33640 1 -- 172.21.15.3:0/1079881588 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e6806dbd0 con 0x7f8e6813c480 2024-03-25T18:28:23.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.200+0000 7f8e64ff9640 1 -- 172.21.15.3:0/1079881588 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e600020e0 con 0x7f8e6813c480 2024-03-25T18:28:23.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.200+0000 7f8e64ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:23.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.200+0000 7f8e64ff9640 1 -- 172.21.15.3:0/1079881588 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f8e600015d0 con 0x7f8e6813c480 2024-03-25T18:28:23.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.200+0000 7f8e64ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:23.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.200+0000 7f8e64ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:23.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.200+0000 7f8e64ff9640 10 monclient: dump: 2024-03-25T18:28:23.201 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:23.201 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:23.201 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:23.201 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:23.201 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:23.201 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:23.201 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:23.201 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:23.201 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:23.201 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:23.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.200+0000 7f8e667fc640 4 set_mon_vals no callback set 2024-03-25T18:28:23.202 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.201+0000 7f8e64ff9640 1 -- 172.21.15.3:0/1079881588 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f8e6001aee0 con 0x7f8e6813c480 2024-03-25T18:28:23.202 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.201+0000 7f8e67fff640 10 monclient: get_auth_request con 0x7f8e5406ddb0 auth_method 0 2024-03-25T18:28:23.202 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.202+0000 7f8e64ff9640 10 client.4355.objecter ms_handle_connect 0x7f8e5406ddb0 2024-03-25T18:28:23.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.207+0000 7f8e64ff9640 1 -- 172.21.15.3:0/1079881588 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f8e60054b80 con 0x7f8e6813c480 2024-03-25T18:28:23.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.207+0000 7f8e64ff9640 10 client.4355.objecter ms_dispatch 0x7f8e68000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:23.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.207+0000 7f8e64ff9640 3 client.4355.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:23.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.207+0000 7f8e64ff9640 3 client.4355.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:23.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.207+0000 7f8e64ff9640 20 client.4355.objecter dump_active .. 0 homeless 2024-03-25T18:28:23.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.207+0000 7f8e64ff9640 1 -- 172.21.15.3:0/1079881588 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8e600504c0 con 0x7f8e6813c480 2024-03-25T18:28:23.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.207+0000 7f8e64ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:23.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.207+0000 7f8e64ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:23.472 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.472+0000 7f8e6ea33640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:23.472 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.472+0000 7f8e6ea33640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:23.472 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.472+0000 7f8e6ea33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:28:23.472 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.472+0000 7f8e6ea33640 1 -- 172.21.15.3:0/1079881588 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e68146780 con 0x7f8e6813c480 2024-03-25T18:28:23.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.503+0000 7f8e64ff9640 1 -- 172.21.15.3:0/1079881588 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f8e68146780 con 0x7f8e6813c480 2024-03-25T18:28:23.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.503+0000 7f8e64ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:23.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.503+0000 7f8e64ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:23.504 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:28:23.504 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:23.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.505+0000 7f8e6ea33640 10 librados: watch_flush enter 2024-03-25T18:28:23.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.505+0000 7f8e6ea33640 10 librados: watch_flush exit 2024-03-25T18:28:23.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.506+0000 7f8e6ea33640 20 client.4355.objecter shutdown clearing up homeless session... 2024-03-25T18:28:23.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.506+0000 7f8e6ea33640 10 client.4355.objecter successfully canceled tick 2024-03-25T18:28:23.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.506+0000 7f8e6ea33640 1 -- 172.21.15.3:0/1079881588 >> v1:172.21.15.3:6800/4195140966 conn(0x7f8e5406ddb0 legacy=0x7f8e54070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:23.507 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.506+0000 7f8e6ea33640 10 monclient: shutdown 2024-03-25T18:28:23.507 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.506+0000 7f8e6ea33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:23.507 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.506+0000 7f8e6ea33640 1 -- 172.21.15.3:0/1079881588 >> v1:172.21.15.3:6790/0 conn(0x7f8e6813c480 legacy=0x7f8e681db840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:23.507 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.506+0000 7f8e6ea33640 1 -- 172.21.15.3:0/1079881588 shutdown_connections 2024-03-25T18:28:23.507 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.506+0000 7f8e6ea33640 1 -- 172.21.15.3:0/1079881588 >> 172.21.15.3:0/1079881588 conn(0x7f8e680617a0 msgr2=0x7f8e680b2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:23.507 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.506+0000 7f8e6ea33640 1 -- 172.21.15.3:0/1079881588 shutdown_connections 2024-03-25T18:28:23.507 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.506+0000 7f8e6ea33640 1 -- 172.21.15.3:0/1079881588 wait complete. 2024-03-25T18:28:23.507 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:23.506+0000 7f8e6ea33640 1 librados: shutdown 2024-03-25T18:28:23.518 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:30.325 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:30.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.437+0000 7f77aa59f640 10 monclient: get_monmap_and_config 2024-03-25T18:28:30.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.437+0000 7f77aa59f640 10 monclient: build_initial_monmap 2024-03-25T18:28:30.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.437+0000 7f77aa59f640 10 monclient: monmap: 2024-03-25T18:28:30.439 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:28:30.439 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:30.439 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:28:30.438980+0000 2024-03-25T18:28:30.439 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:28:30.438980+0000 2024-03-25T18:28:30.439 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:30.439 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:30.440 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:28:30.440 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:28:30.440 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:28:30.440 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:30.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.438+0000 7f77aa59f640 1 Processor -- start 2024-03-25T18:28:30.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.438+0000 7f77aa59f640 1 -- start start 2024-03-25T18:28:30.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.438+0000 7f77aa59f640 10 monclient: init 2024-03-25T18:28:30.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.439+0000 7f77aa59f640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:30.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.439+0000 7f77aa59f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:28:30.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.439+0000 7f77aa59f640 10 monclient(hunting): picked mon.noname-c con 0x7f77a40b4e40 addr v1:172.21.15.16:6789/0 2024-03-25T18:28:30.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.439+0000 7f77aa59f640 10 monclient(hunting): picked mon.noname-b con 0x7f77a40a9660 addr v1:172.21.15.3:6790/0 2024-03-25T18:28:30.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.439+0000 7f77aa59f640 10 monclient(hunting): picked mon.noname-a con 0x7f77a4145f70 addr v1:172.21.15.3:6789/0 2024-03-25T18:28:30.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.439+0000 7f77aa59f640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77a40ab2e0 con 0x7f77a4145f70 2024-03-25T18:28:30.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.439+0000 7f77aa59f640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77a40a9c10 con 0x7f77a40b4e40 2024-03-25T18:28:30.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.439+0000 7f77aa59f640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77a413fee0 con 0x7f77a40a9660 2024-03-25T18:28:30.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.439+0000 7f77aa59f640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:30.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.439+0000 7f77aa59f640 10 monclient(hunting): authenticate will time out at 7564.699987s 2024-03-25T18:28:30.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.445+0000 7f77a9d9e640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f77a4145f70 0x7f77a414dbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52166/0 (socket says 172.21.15.3:52166) 2024-03-25T18:28:30.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.445+0000 7f77a9d9e640 1 -- 172.21.15.3:0/465448008 learned_addr learned my addr 172.21.15.3:0/465448008 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:28:30.462 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.461+0000 7f779bfff640 1 -- 172.21.15.3:0/465448008 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635226590 0 0) 0x7f77a40ab2e0 con 0x7f77a4145f70 2024-03-25T18:28:30.462 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.461+0000 7f779bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:30.462 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.461+0000 7f779bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:30.462 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.461+0000 7f779bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:30.462 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.461+0000 7f779bfff640 10 monclient(hunting): my global_id is 4389 2024-03-25T18:28:30.462 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.461+0000 7f779bfff640 1 -- 172.21.15.3:0/465448008 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7790003520 con 0x7f77a4145f70 2024-03-25T18:28:30.473 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.471+0000 7f779bfff640 1 -- 172.21.15.3:0/465448008 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1995803828 0 0) 0x7f7790003520 con 0x7f77a4145f70 2024-03-25T18:28:30.473 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.471+0000 7f779bfff640 1 -- 172.21.15.3:0/465448008 >> v1:172.21.15.3:6790/0 conn(0x7f77a40a9660 legacy=0x7f77a4145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:30.473 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.471+0000 7f779bfff640 1 -- 172.21.15.3:0/465448008 >> v1:172.21.15.16:6789/0 conn(0x7f77a40b4e40 legacy=0x7f77a413f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:30.473 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.471+0000 7f779bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:28:30.473 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.471+0000 7f779bfff640 1 monclient: found mon.noname-a 2024-03-25T18:28:30.473 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.471+0000 7f779bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:30.473 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.471+0000 7f779bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:28:30.474 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.471+0000 7f779bfff640 1 -- 172.21.15.3:0/465448008 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a41463c0 con 0x7f77a4145f70 2024-03-25T18:28:30.474 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.471+0000 7f779bfff640 10 monclient: _finish_auth 0 2024-03-25T18:28:30.474 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.471+0000 7f779bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:28:30.474 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.471+0000 7f779bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:30.474 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.471+0000 7f779bfff640 1 -- 172.21.15.3:0/465448008 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f779c001a10 con 0x7f77a4145f70 2024-03-25T18:28:30.474 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.471+0000 7f779bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:30.474 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.471+0000 7f779bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:30.474 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.471+0000 7f779bfff640 10 monclient: dump: 2024-03-25T18:28:30.474 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:30.474 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:30.474 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:30.474 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:30.474 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:30.474 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:30.475 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:30.475 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:30.475 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:30.475 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:30.475 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.471+0000 7f77aa59f640 5 monclient: authenticate success, global_id 4389 2024-03-25T18:28:30.475 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.471+0000 7f77aa59f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:30.490 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.489+0000 7f779bfff640 1 -- 172.21.15.3:0/465448008 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f779c001b70 con 0x7f77a4145f70 2024-03-25T18:28:30.490 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.489+0000 7f779bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:30.490 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.489+0000 7f779bfff640 1 -- 172.21.15.3:0/465448008 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f779c001f80 con 0x7f77a4145f70 2024-03-25T18:28:30.490 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.489+0000 7f77aa59f640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:30.490 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.489+0000 7f77aa59f640 4 set_mon_vals no callback set 2024-03-25T18:28:30.490 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.489+0000 7f779bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:30.490 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.489+0000 7f779bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:30.491 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.489+0000 7f779bfff640 10 monclient: dump: 2024-03-25T18:28:30.491 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:30.491 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:30.491 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:30.491 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:30.491 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:30.491 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:30.491 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:30.491 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:30.491 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:30.491 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:30.493 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.489+0000 7f77aa59f640 10 monclient: shutdown 2024-03-25T18:28:30.493 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.489+0000 7f77aa59f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:30.493 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.489+0000 7f77aa59f640 1 -- 172.21.15.3:0/465448008 >> v1:172.21.15.3:6789/0 conn(0x7f77a4145f70 legacy=0x7f77a414dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:30.493 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.490+0000 7f77aa59f640 1 -- 172.21.15.3:0/465448008 shutdown_connections 2024-03-25T18:28:30.493 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.490+0000 7f77aa59f640 1 -- 172.21.15.3:0/465448008 >> 172.21.15.3:0/465448008 conn(0x7f77a40617a0 msgr2=0x7f77a4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:30.493 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.490+0000 7f77aa59f640 1 -- 172.21.15.3:0/465448008 shutdown_connections 2024-03-25T18:28:30.493 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.490+0000 7f77aa59f640 1 -- 172.21.15.3:0/465448008 wait complete. 2024-03-25T18:28:30.494 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.491+0000 7f77aa59f640 10 monclient: build_initial_monmap 2024-03-25T18:28:30.494 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.491+0000 7f77aa59f640 10 monclient: monmap: 2024-03-25T18:28:30.494 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:28:30.494 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:30.494 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:28:30.494 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:28:30.494 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:30.494 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:28:30.494 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:28:30.494 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:28:30.494 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:28:30.494 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:30.495 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.491+0000 7f77aa59f640 1 librados: starting msgr at 2024-03-25T18:28:30.495 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.491+0000 7f77aa59f640 1 librados: starting objecter 2024-03-25T18:28:30.495 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.491+0000 7f77aa59f640 1 Processor -- start 2024-03-25T18:28:30.495 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.491+0000 7f77aa59f640 1 -- start start 2024-03-25T18:28:30.495 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.491+0000 7f77aa59f640 1 librados: setting wanted keys 2024-03-25T18:28:30.495 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.491+0000 7f77aa59f640 1 librados: calling monclient init 2024-03-25T18:28:30.495 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.491+0000 7f77aa59f640 10 monclient: init 2024-03-25T18:28:30.495 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.492+0000 7f77aa59f640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:30.495 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.492+0000 7f77aa59f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:28:30.495 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.492+0000 7f77aa59f640 10 monclient(hunting): picked mon.noname-a con 0x7f77a40a9660 addr v1:172.21.15.3:6789/0 2024-03-25T18:28:30.495 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.492+0000 7f77aa59f640 10 monclient(hunting): picked mon.noname-c con 0x7f77a40b4e40 addr v1:172.21.15.3:6790/0 2024-03-25T18:28:30.495 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.492+0000 7f77aa59f640 10 monclient(hunting): picked mon.noname-b con 0x7f77a4145f70 addr v1:172.21.15.16:6789/0 2024-03-25T18:28:30.495 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.492+0000 7f77aa59f640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77a41366c0 con 0x7f77a40a9660 2024-03-25T18:28:30.495 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.492+0000 7f77aa59f640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77a41369a0 con 0x7f77a4145f70 2024-03-25T18:28:30.496 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.492+0000 7f77aa59f640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77a4140be0 con 0x7f77a40b4e40 2024-03-25T18:28:30.496 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.492+0000 7f77aa59f640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:30.496 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.492+0000 7f77aa59f640 10 monclient(hunting): authenticate will time out at 7564.752727s 2024-03-25T18:28:30.508 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.507+0000 7f77a9d9e640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f77a4145f70 0x7f77a41e2bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:58792/0 (socket says 172.21.15.3:58792) 2024-03-25T18:28:30.508 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.507+0000 7f77a9d9e640 1 -- 172.21.15.3:0/2731276636 learned_addr learned my addr 172.21.15.3:0/2731276636 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:28:30.513 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.512+0000 7f7799ffb640 10 client.?.objecter ms_handle_connect 0x7f77a40b4e40 2024-03-25T18:28:30.513 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.512+0000 7f7799ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:30.513 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.513+0000 7f7799ffb640 10 client.?.objecter ms_handle_connect 0x7f77a4145f70 2024-03-25T18:28:30.514 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.513+0000 7f7799ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:30.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.516+0000 7f7799ffb640 1 -- 172.21.15.3:0/2731276636 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 997012610 0 0) 0x7f77a4140be0 con 0x7f77a40b4e40 2024-03-25T18:28:30.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.516+0000 7f7799ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:28:30.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.516+0000 7f7799ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:30.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.516+0000 7f7799ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:30.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.516+0000 7f7799ffb640 10 monclient(hunting): my global_id is 4361 2024-03-25T18:28:30.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.516+0000 7f7799ffb640 1 -- 172.21.15.3:0/2731276636 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7788004eb0 con 0x7f77a40b4e40 2024-03-25T18:28:30.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.522+0000 7f7799ffb640 1 -- 172.21.15.3:0/2731276636 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2994765327 0 0) 0x7f77a41369a0 con 0x7f77a4145f70 2024-03-25T18:28:30.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.522+0000 7f7799ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:28:30.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.522+0000 7f7799ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:30.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.522+0000 7f7799ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:30.524 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.522+0000 7f7799ffb640 10 monclient(hunting): my global_id is 4354 2024-03-25T18:28:30.524 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.522+0000 7f7799ffb640 1 -- 172.21.15.3:0/2731276636 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77a4140be0 con 0x7f77a4145f70 2024-03-25T18:28:30.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.530+0000 7f7799ffb640 10 client.?.objecter ms_handle_connect 0x7f77a40a9660 2024-03-25T18:28:30.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.530+0000 7f7799ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:30.542 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.541+0000 7f7799ffb640 1 -- 172.21.15.3:0/2731276636 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 309867055 0 0) 0x7f7788004eb0 con 0x7f77a40b4e40 2024-03-25T18:28:30.542 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.541+0000 7f7799ffb640 1 -- 172.21.15.3:0/2731276636 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77a41369a0 con 0x7f77a40b4e40 2024-03-25T18:28:30.542 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.541+0000 7f7799ffb640 1 -- 172.21.15.3:0/2731276636 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7794001930 con 0x7f77a40b4e40 2024-03-25T18:28:30.542 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.541+0000 7f7799ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:30.542 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.541+0000 7f7799ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:30.542 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.541+0000 7f7799ffb640 10 monclient(hunting): dump: 2024-03-25T18:28:30.542 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:30.542 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:30.543 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:30.543 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:30.543 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:30.543 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:30.543 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:30.543 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:30.543 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:30.543 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:30.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.541+0000 7f7799ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:30.545 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.543+0000 7f7799ffb640 1 -- 172.21.15.3:0/2731276636 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1733570904 0 0) 0x7f77a4140be0 con 0x7f77a4145f70 2024-03-25T18:28:30.545 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.543+0000 7f7799ffb640 1 -- 172.21.15.3:0/2731276636 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7788004eb0 con 0x7f77a4145f70 2024-03-25T18:28:30.545 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.543+0000 7f7799ffb640 1 -- 172.21.15.3:0/2731276636 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f779c001f40 con 0x7f77a4145f70 2024-03-25T18:28:30.545 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.543+0000 7f7799ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:30.545 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.543+0000 7f7799ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:30.545 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.543+0000 7f7799ffb640 10 monclient(hunting): dump: 2024-03-25T18:28:30.545 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:30.545 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:30.546 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:30.546 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:30.546 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:30.546 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:30.546 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:30.546 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:30.546 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:30.546 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:30.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.556+0000 7f7799ffb640 1 -- 172.21.15.3:0/2731276636 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779903833 0 0) 0x7f77a41366c0 con 0x7f77a40a9660 2024-03-25T18:28:30.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.556+0000 7f7799ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:28:30.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.556+0000 7f7799ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:30.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.556+0000 7f7799ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:30.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.556+0000 7f7799ffb640 10 monclient(hunting): my global_id is 4392 2024-03-25T18:28:30.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.556+0000 7f7799ffb640 1 -- 172.21.15.3:0/2731276636 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77a4140be0 con 0x7f77a40a9660 2024-03-25T18:28:30.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.558+0000 7f7799ffb640 1 -- 172.21.15.3:0/2731276636 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1290537293 0 0) 0x7f7788004eb0 con 0x7f77a4145f70 2024-03-25T18:28:30.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.558+0000 7f7799ffb640 1 -- 172.21.15.3:0/2731276636 >> v1:172.21.15.3:6790/0 conn(0x7f77a40b4e40 legacy=0x7f77a4141100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:30.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.558+0000 7f7799ffb640 1 -- 172.21.15.3:0/2731276636 >> v1:172.21.15.3:6789/0 conn(0x7f77a40a9660 legacy=0x7f77a41404d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:30.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.558+0000 7f7799ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:28:30.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.558+0000 7f7799ffb640 1 monclient: found mon.b 2024-03-25T18:28:30.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.558+0000 7f7799ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:30.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.558+0000 7f7799ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:28:30.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.558+0000 7f7799ffb640 1 -- 172.21.15.3:0/2731276636 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a4134e00 con 0x7f77a4145f70 2024-03-25T18:28:30.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.558+0000 7f7799ffb640 10 monclient: _finish_auth 0 2024-03-25T18:28:30.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.558+0000 7f7799ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:28:30.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.558+0000 7f7799ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:30.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.558+0000 7f77aa59f640 5 monclient: authenticate success, global_id 4354 2024-03-25T18:28:30.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.558+0000 7f77aa59f640 10 monclient: _renew_subs 2024-03-25T18:28:30.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.558+0000 7f77aa59f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:28:30.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.558+0000 7f77aa59f640 1 -- 172.21.15.3:0/2731276636 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77a4142e60 con 0x7f77a4145f70 2024-03-25T18:28:30.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.558+0000 7f77aa59f640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:30.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.558+0000 7f77aa59f640 10 monclient: _renew_subs 2024-03-25T18:28:30.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.558+0000 7f77aa59f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:28:30.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.558+0000 7f77aa59f640 1 -- 172.21.15.3:0/2731276636 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f77a41db620 con 0x7f77a4145f70 2024-03-25T18:28:30.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.558+0000 7f77aa59f640 1 librados: init done 2024-03-25T18:28:30.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.559+0000 7f77aa59f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:30.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.559+0000 7f77aa59f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:30.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.559+0000 7f77aa59f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:28:30.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.559+0000 7f77aa59f640 1 -- 172.21.15.3:0/2731276636 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77a4134ff0 con 0x7f77a4145f70 2024-03-25T18:28:30.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.842+0000 7f7799ffb640 1 -- 172.21.15.3:0/2731276636 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f779c004e10 con 0x7f77a4145f70 2024-03-25T18:28:30.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.842+0000 7f7799ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:30.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.842+0000 7f7799ffb640 1 -- 172.21.15.3:0/2731276636 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f779c005120 con 0x7f77a4145f70 2024-03-25T18:28:30.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.842+0000 7f7799ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:30.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.842+0000 7f7799ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:30.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.842+0000 7f7799ffb640 10 monclient: dump: 2024-03-25T18:28:30.844 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:30.845 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:30.845 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:30.845 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:30.845 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:30.845 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:30.845 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:30.845 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:30.845 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:30.845 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:30.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.842+0000 7f779b7fe640 4 set_mon_vals no callback set 2024-03-25T18:28:30.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.844+0000 7f7799ffb640 1 -- 172.21.15.3:0/2731276636 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f779c01b0f0 con 0x7f77a4145f70 2024-03-25T18:28:30.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.844+0000 7f77a959d640 10 monclient: get_auth_request con 0x7f778806df30 auth_method 0 2024-03-25T18:28:30.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.845+0000 7f7799ffb640 10 client.4354.objecter ms_handle_connect 0x7f778806df30 2024-03-25T18:28:30.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.849+0000 7f7799ffb640 1 -- 172.21.15.3:0/2731276636 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f779c054b90 con 0x7f77a4145f70 2024-03-25T18:28:30.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.849+0000 7f7799ffb640 10 client.4354.objecter ms_dispatch 0x7f77a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:30.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.849+0000 7f7799ffb640 3 client.4354.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:30.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.849+0000 7f7799ffb640 3 client.4354.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:30.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.849+0000 7f7799ffb640 20 client.4354.objecter dump_active .. 0 homeless 2024-03-25T18:28:30.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.849+0000 7f7799ffb640 1 -- 172.21.15.3:0/2731276636 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f779c0506a0 con 0x7f77a4145f70 2024-03-25T18:28:30.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.849+0000 7f7799ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:30.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:30.849+0000 7f7799ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:31.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:31.097+0000 7f77aa59f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:31.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:31.097+0000 7f77aa59f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:31.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:31.097+0000 7f77aa59f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:28:31.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:31.097+0000 7f77aa59f640 1 -- 172.21.15.3:0/2731276636 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77a4068030 con 0x7f77a4145f70 2024-03-25T18:28:31.136 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:31.135+0000 7f7799ffb640 1 -- 172.21.15.3:0/2731276636 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f779c0525d0 con 0x7f77a4145f70 2024-03-25T18:28:31.136 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:31.135+0000 7f7799ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:31.136 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:31.135+0000 7f7799ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:31.136 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:28:31.137 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:31.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:31.137+0000 7f77aa59f640 10 librados: watch_flush enter 2024-03-25T18:28:31.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:31.137+0000 7f77aa59f640 10 librados: watch_flush exit 2024-03-25T18:28:31.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:31.137+0000 7f77aa59f640 20 client.4354.objecter shutdown clearing up homeless session... 2024-03-25T18:28:31.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:31.137+0000 7f77aa59f640 10 client.4354.objecter successfully canceled tick 2024-03-25T18:28:31.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:31.137+0000 7f77aa59f640 1 -- 172.21.15.3:0/2731276636 >> v1:172.21.15.3:6800/4195140966 conn(0x7f778806df30 legacy=0x7f77880703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:31.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:31.137+0000 7f77aa59f640 10 monclient: shutdown 2024-03-25T18:28:31.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:31.137+0000 7f77aa59f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:31.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:31.137+0000 7f77aa59f640 1 -- 172.21.15.3:0/2731276636 >> v1:172.21.15.16:6789/0 conn(0x7f77a4145f70 legacy=0x7f77a41e2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:31.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:31.138+0000 7f77aa59f640 1 -- 172.21.15.3:0/2731276636 shutdown_connections 2024-03-25T18:28:31.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:31.138+0000 7f77aa59f640 1 -- 172.21.15.3:0/2731276636 >> 172.21.15.3:0/2731276636 conn(0x7f77a40617a0 msgr2=0x7f77a413b920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:31.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:31.138+0000 7f77aa59f640 1 -- 172.21.15.3:0/2731276636 shutdown_connections 2024-03-25T18:28:31.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:31.138+0000 7f77aa59f640 1 -- 172.21.15.3:0/2731276636 wait complete. 2024-03-25T18:28:31.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:31.138+0000 7f77aa59f640 1 librados: shutdown 2024-03-25T18:28:31.150 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:37.956 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:38.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.064+0000 7f15ee7cf640 10 monclient: get_monmap_and_config 2024-03-25T18:28:38.066 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.064+0000 7f15ee7cf640 10 monclient: build_initial_monmap 2024-03-25T18:28:38.066 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.064+0000 7f15ee7cf640 10 monclient: monmap: 2024-03-25T18:28:38.066 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:28:38.066 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:38.066 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:28:38.065907+0000 2024-03-25T18:28:38.066 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:28:38.065907+0000 2024-03-25T18:28:38.066 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:38.066 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:38.066 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:28:38.066 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:28:38.066 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:28:38.066 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:38.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.065+0000 7f15ee7cf640 1 Processor -- start 2024-03-25T18:28:38.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.065+0000 7f15ee7cf640 1 -- start start 2024-03-25T18:28:38.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.065+0000 7f15ee7cf640 10 monclient: init 2024-03-25T18:28:38.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.066+0000 7f15ee7cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:38.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.066+0000 7f15ee7cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:38.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.066+0000 7f15ee7cf640 10 monclient(hunting): picked mon.noname-b con 0x7f15e80b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:28:38.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.066+0000 7f15ee7cf640 10 monclient(hunting): picked mon.noname-c con 0x7f15e80b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:28:38.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.066+0000 7f15ee7cf640 10 monclient(hunting): picked mon.noname-a con 0x7f15e813c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:28:38.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.066+0000 7f15ee7cf640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15e80ab0e0 con 0x7f15e813c480 2024-03-25T18:28:38.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.066+0000 7f15ee7cf640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15e80ab3c0 con 0x7f15e80b4670 2024-03-25T18:28:38.069 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.066+0000 7f15ee7cf640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15e80a9f30 con 0x7f15e80b4ff0 2024-03-25T18:28:38.069 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.066+0000 7f15ee7cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:38.069 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.066+0000 7f15ee7cf640 10 monclient(hunting): authenticate will time out at 7572.326968s 2024-03-25T18:28:38.069 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.067+0000 7f15e7fff640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f15e80b4ff0 0x7f15e813bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:32790/0 (socket says 172.21.15.3:32790) 2024-03-25T18:28:38.069 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.067+0000 7f15e7fff640 1 -- 172.21.15.3:0/3420119676 learned_addr learned my addr 172.21.15.3:0/3420119676 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:28:38.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.123+0000 7f15e6ffd640 1 -- 172.21.15.3:0/3420119676 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1926185999 0 0) 0x7f15e80ab3c0 con 0x7f15e80b4670 2024-03-25T18:28:38.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.123+0000 7f15e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:38.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.123+0000 7f15e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:38.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.123+0000 7f15e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:38.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.123+0000 7f15e6ffd640 10 monclient(hunting): my global_id is 4357 2024-03-25T18:28:38.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.123+0000 7f15e6ffd640 1 -- 172.21.15.3:0/3420119676 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15d0003520 con 0x7f15e80b4670 2024-03-25T18:28:38.127 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.126+0000 7f15e6ffd640 1 -- 172.21.15.3:0/3420119676 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1946343402 0 0) 0x7f15e80a9f30 con 0x7f15e80b4ff0 2024-03-25T18:28:38.127 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.126+0000 7f15e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:38.127 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.126+0000 7f15e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:38.127 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.126+0000 7f15e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:38.127 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.126+0000 7f15e6ffd640 10 monclient(hunting): my global_id is 4364 2024-03-25T18:28:38.127 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.126+0000 7f15e6ffd640 1 -- 172.21.15.3:0/3420119676 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15e80ab3c0 con 0x7f15e80b4ff0 2024-03-25T18:28:38.138 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.137+0000 7f15e6ffd640 1 -- 172.21.15.3:0/3420119676 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1272117018 0 0) 0x7f15e80ab0e0 con 0x7f15e813c480 2024-03-25T18:28:38.138 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.137+0000 7f15e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:38.138 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.137+0000 7f15e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:38.138 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.137+0000 7f15e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:38.138 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.137+0000 7f15e6ffd640 10 monclient(hunting): my global_id is 4395 2024-03-25T18:28:38.138 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.137+0000 7f15e6ffd640 1 -- 172.21.15.3:0/3420119676 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15e80a9f30 con 0x7f15e813c480 2024-03-25T18:28:38.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.138+0000 7f15e6ffd640 1 -- 172.21.15.3:0/3420119676 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3604088901 0 0) 0x7f15d0003520 con 0x7f15e80b4670 2024-03-25T18:28:38.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.138+0000 7f15e6ffd640 1 -- 172.21.15.3:0/3420119676 >> v1:172.21.15.3:6790/0 conn(0x7f15e80b4ff0 legacy=0x7f15e813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:38.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.138+0000 7f15e6ffd640 1 -- 172.21.15.3:0/3420119676 >> v1:172.21.15.3:6789/0 conn(0x7f15e813c480 legacy=0x7f15e814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:38.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.138+0000 7f15e6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:28:38.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.138+0000 7f15e6ffd640 1 monclient: found mon.noname-c 2024-03-25T18:28:38.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.138+0000 7f15e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:38.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.138+0000 7f15e6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:28:38.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.138+0000 7f15e6ffd640 1 -- 172.21.15.3:0/3420119676 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e80a9b70 con 0x7f15e80b4670 2024-03-25T18:28:38.141 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.138+0000 7f15e6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:28:38.141 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.138+0000 7f15e6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:28:38.141 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.138+0000 7f15e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:38.141 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.138+0000 7f15e6ffd640 1 -- 172.21.15.3:0/3420119676 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f15dc001690 con 0x7f15e80b4670 2024-03-25T18:28:38.141 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.138+0000 7f15e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:38.141 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.138+0000 7f15e6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:38.141 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.138+0000 7f15e6ffd640 10 monclient: dump: 2024-03-25T18:28:38.141 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:38.141 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:38.141 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:38.141 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:38.141 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:38.141 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:38.142 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:38.142 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:38.142 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:38.142 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:38.142 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.138+0000 7f15ee7cf640 5 monclient: authenticate success, global_id 4357 2024-03-25T18:28:38.142 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.138+0000 7f15ee7cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:38.151 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.150+0000 7f15e6ffd640 1 -- 172.21.15.3:0/3420119676 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15dc0017f0 con 0x7f15e80b4670 2024-03-25T18:28:38.151 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.150+0000 7f15e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:38.151 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.150+0000 7f15ee7cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:38.151 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.150+0000 7f15ee7cf640 4 set_mon_vals no callback set 2024-03-25T18:28:38.152 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.150+0000 7f15ee7cf640 10 monclient: shutdown 2024-03-25T18:28:38.152 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.150+0000 7f15ee7cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:38.152 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.150+0000 7f15ee7cf640 1 -- 172.21.15.3:0/3420119676 >> v1:172.21.15.16:6789/0 conn(0x7f15e80b4670 legacy=0x7f15e81425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:38.152 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.151+0000 7f15ee7cf640 1 -- 172.21.15.3:0/3420119676 shutdown_connections 2024-03-25T18:28:38.152 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.151+0000 7f15ee7cf640 1 -- 172.21.15.3:0/3420119676 >> 172.21.15.3:0/3420119676 conn(0x7f15e80617a0 msgr2=0x7f15e8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:38.152 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.151+0000 7f15ee7cf640 1 -- 172.21.15.3:0/3420119676 shutdown_connections 2024-03-25T18:28:38.152 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.151+0000 7f15ee7cf640 1 -- 172.21.15.3:0/3420119676 wait complete. 2024-03-25T18:28:38.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.151+0000 7f15ee7cf640 10 monclient: build_initial_monmap 2024-03-25T18:28:38.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.151+0000 7f15ee7cf640 10 monclient: monmap: 2024-03-25T18:28:38.153 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:28:38.153 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:38.153 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:28:38.153 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:28:38.153 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:38.153 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:28:38.153 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:28:38.154 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:28:38.154 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:28:38.154 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:38.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.152+0000 7f15ee7cf640 1 librados: starting msgr at 2024-03-25T18:28:38.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.152+0000 7f15ee7cf640 1 librados: starting objecter 2024-03-25T18:28:38.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.152+0000 7f15ee7cf640 1 Processor -- start 2024-03-25T18:28:38.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.152+0000 7f15ee7cf640 1 -- start start 2024-03-25T18:28:38.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.152+0000 7f15ee7cf640 1 librados: setting wanted keys 2024-03-25T18:28:38.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.152+0000 7f15ee7cf640 1 librados: calling monclient init 2024-03-25T18:28:38.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.152+0000 7f15ee7cf640 10 monclient: init 2024-03-25T18:28:38.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.153+0000 7f15ee7cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:38.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.153+0000 7f15ee7cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:38.156 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.153+0000 7f15ee7cf640 10 monclient(hunting): picked mon.noname-c con 0x7f15e80b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:28:38.156 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.153+0000 7f15ee7cf640 10 monclient(hunting): picked mon.noname-b con 0x7f15e80b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:28:38.156 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.153+0000 7f15ee7cf640 10 monclient(hunting): picked mon.noname-a con 0x7f15e813c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:28:38.156 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.153+0000 7f15ee7cf640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15e8144d40 con 0x7f15e813c480 2024-03-25T18:28:38.156 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.153+0000 7f15ee7cf640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15e813c0f0 con 0x7f15e80b4ff0 2024-03-25T18:28:38.156 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.153+0000 7f15ee7cf640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15e813d7e0 con 0x7f15e80b4670 2024-03-25T18:28:38.156 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.153+0000 7f15ee7cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:38.156 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.153+0000 7f15ee7cf640 10 monclient(hunting): authenticate will time out at 7572.413862s 2024-03-25T18:28:38.162 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.161+0000 7f15e7fff640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f15e80b4670 0x7f15e813e810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:32810/0 (socket says 172.21.15.3:32810) 2024-03-25T18:28:38.162 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.161+0000 7f15e7fff640 1 -- 172.21.15.3:0/2471921486 learned_addr learned my addr 172.21.15.3:0/2471921486 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:28:38.164 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.163+0000 7f15e4ff9640 10 client.?.objecter ms_handle_connect 0x7f15e80b4670 2024-03-25T18:28:38.164 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.163+0000 7f15e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:38.174 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.173+0000 7f15e4ff9640 10 client.?.objecter ms_handle_connect 0x7f15e80b4ff0 2024-03-25T18:28:38.174 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.173+0000 7f15e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:38.176 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.175+0000 7f15e4ff9640 1 -- 172.21.15.3:0/2471921486 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 203507452 0 0) 0x7f15e813d7e0 con 0x7f15e80b4670 2024-03-25T18:28:38.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.175+0000 7f15e4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:38.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.175+0000 7f15e4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:38.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.175+0000 7f15e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:38.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.175+0000 7f15e4ff9640 10 monclient(hunting): my global_id is 4367 2024-03-25T18:28:38.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.175+0000 7f15e4ff9640 1 -- 172.21.15.3:0/2471921486 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15d4004eb0 con 0x7f15e80b4670 2024-03-25T18:28:38.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.179+0000 7f15e4ff9640 10 client.?.objecter ms_handle_connect 0x7f15e813c480 2024-03-25T18:28:38.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.179+0000 7f15e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:38.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.191+0000 7f15e4ff9640 1 -- 172.21.15.3:0/2471921486 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4188530234 0 0) 0x7f15e813c0f0 con 0x7f15e80b4ff0 2024-03-25T18:28:38.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.191+0000 7f15e4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:38.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.191+0000 7f15e4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:38.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.191+0000 7f15e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:38.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.191+0000 7f15e4ff9640 10 monclient(hunting): my global_id is 4360 2024-03-25T18:28:38.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.191+0000 7f15e4ff9640 1 -- 172.21.15.3:0/2471921486 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15e813d7e0 con 0x7f15e80b4ff0 2024-03-25T18:28:38.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.195+0000 7f15e4ff9640 1 -- 172.21.15.3:0/2471921486 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454960794 0 0) 0x7f15e80a9f30 con 0x7f15e813c480 2024-03-25T18:28:38.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.195+0000 7f15e4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:38.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.195+0000 7f15e4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:38.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.195+0000 7f15e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:38.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.195+0000 7f15e4ff9640 10 monclient(hunting): my global_id is 4398 2024-03-25T18:28:38.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.195+0000 7f15e4ff9640 1 -- 172.21.15.3:0/2471921486 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15e813c0f0 con 0x7f15e813c480 2024-03-25T18:28:38.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.196+0000 7f15e4ff9640 1 -- 172.21.15.3:0/2471921486 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1836604672 0 0) 0x7f15d4004eb0 con 0x7f15e80b4670 2024-03-25T18:28:38.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.196+0000 7f15e4ff9640 1 -- 172.21.15.3:0/2471921486 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15e80a9f30 con 0x7f15e80b4670 2024-03-25T18:28:38.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.196+0000 7f15e4ff9640 1 -- 172.21.15.3:0/2471921486 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f15d8001970 con 0x7f15e80b4670 2024-03-25T18:28:38.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.196+0000 7f15e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:38.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.196+0000 7f15e4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:38.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.196+0000 7f15e4ff9640 10 monclient(hunting): dump: 2024-03-25T18:28:38.198 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:38.198 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:38.198 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:38.198 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:38.198 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:38.198 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:38.198 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:38.198 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:38.198 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:38.198 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:38.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.196+0000 7f15e4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:38.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.216+0000 7f15e4ff9640 1 -- 172.21.15.3:0/2471921486 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3594701318 0 0) 0x7f15e813d7e0 con 0x7f15e80b4ff0 2024-03-25T18:28:38.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.216+0000 7f15e4ff9640 1 -- 172.21.15.3:0/2471921486 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15d4004eb0 con 0x7f15e80b4ff0 2024-03-25T18:28:38.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.216+0000 7f15e4ff9640 1 -- 172.21.15.3:0/2471921486 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f15dc001ca0 con 0x7f15e80b4ff0 2024-03-25T18:28:38.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.216+0000 7f15e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:38.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.216+0000 7f15e4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:38.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.216+0000 7f15e4ff9640 10 monclient(hunting): dump: 2024-03-25T18:28:38.218 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:38.218 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:38.218 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:38.218 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:38.219 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:38.219 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:38.219 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:38.219 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:38.219 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:38.219 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:38.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.218+0000 7f15e4ff9640 1 -- 172.21.15.3:0/2471921486 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1415030027 0 0) 0x7f15e80a9f30 con 0x7f15e80b4670 2024-03-25T18:28:38.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.218+0000 7f15e4ff9640 1 -- 172.21.15.3:0/2471921486 >> v1:172.21.15.16:6789/0 conn(0x7f15e80b4ff0 legacy=0x7f15e8143620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:38.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.218+0000 7f15e4ff9640 1 -- 172.21.15.3:0/2471921486 >> v1:172.21.15.3:6789/0 conn(0x7f15e813c480 legacy=0x7f15e81e2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:38.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.218+0000 7f15e4ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:28:38.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.218+0000 7f15e4ff9640 1 monclient: found mon.c 2024-03-25T18:28:38.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.218+0000 7f15e4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:38.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.218+0000 7f15e4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:28:38.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.218+0000 7f15e4ff9640 1 -- 172.21.15.3:0/2471921486 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e813dac0 con 0x7f15e80b4670 2024-03-25T18:28:38.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.218+0000 7f15e4ff9640 10 monclient: _finish_auth 0 2024-03-25T18:28:38.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.218+0000 7f15e4ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:28:38.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.218+0000 7f15e4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:38.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.218+0000 7f15ee7cf640 5 monclient: authenticate success, global_id 4367 2024-03-25T18:28:38.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.218+0000 7f15ee7cf640 10 monclient: _renew_subs 2024-03-25T18:28:38.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.218+0000 7f15ee7cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:28:38.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.218+0000 7f15ee7cf640 1 -- 172.21.15.3:0/2471921486 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15e813dda0 con 0x7f15e80b4670 2024-03-25T18:28:38.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.218+0000 7f15ee7cf640 10 client.4367.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:38.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.218+0000 7f15ee7cf640 10 monclient: _renew_subs 2024-03-25T18:28:38.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.218+0000 7f15ee7cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:28:38.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.218+0000 7f15ee7cf640 1 -- 172.21.15.3:0/2471921486 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15e813cd80 con 0x7f15e80b4670 2024-03-25T18:28:38.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.218+0000 7f15ee7cf640 1 librados: init done 2024-03-25T18:28:38.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.219+0000 7f15ee7cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:38.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.219+0000 7f15ee7cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:38.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.219+0000 7f15ee7cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:28:38.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.219+0000 7f15ee7cf640 1 -- 172.21.15.3:0/2471921486 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15e806dbd0 con 0x7f15e80b4670 2024-03-25T18:28:38.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.539+0000 7f15e4ff9640 1 -- 172.21.15.3:0/2471921486 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15d80047e0 con 0x7f15e80b4670 2024-03-25T18:28:38.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.539+0000 7f15e4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:38.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.539+0000 7f15e4ff9640 1 -- 172.21.15.3:0/2471921486 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f15d8004af0 con 0x7f15e80b4670 2024-03-25T18:28:38.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.539+0000 7f15e4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:38.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.539+0000 7f15e4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:38.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.539+0000 7f15e4ff9640 10 monclient: dump: 2024-03-25T18:28:38.541 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:38.541 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:38.541 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:38.541 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:38.541 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:38.541 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:38.542 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:38.542 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:38.542 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:38.542 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:38.542 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.539+0000 7f15e67fc640 4 set_mon_vals no callback set 2024-03-25T18:28:38.542 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.540+0000 7f15e4ff9640 1 -- 172.21.15.3:0/2471921486 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f15d801ab00 con 0x7f15e80b4670 2024-03-25T18:28:38.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.541+0000 7f15e77fe640 10 monclient: get_auth_request con 0x7f15d406dec0 auth_method 0 2024-03-25T18:28:38.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.541+0000 7f15e4ff9640 10 client.4367.objecter ms_handle_connect 0x7f15d406dec0 2024-03-25T18:28:38.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.547+0000 7f15e4ff9640 1 -- 172.21.15.3:0/2471921486 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f15d80547c0 con 0x7f15e80b4670 2024-03-25T18:28:38.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.547+0000 7f15e4ff9640 10 client.4367.objecter ms_dispatch 0x7f15e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:38.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.547+0000 7f15e4ff9640 3 client.4367.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:38.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.547+0000 7f15e4ff9640 3 client.4367.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:38.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.547+0000 7f15e4ff9640 20 client.4367.objecter dump_active .. 0 homeless 2024-03-25T18:28:38.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.547+0000 7f15e4ff9640 1 -- 172.21.15.3:0/2471921486 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f15d8050100 con 0x7f15e80b4670 2024-03-25T18:28:38.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.547+0000 7f15e4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:38.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.547+0000 7f15e4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:38.806 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.805+0000 7f15ee7cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:38.806 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.805+0000 7f15ee7cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:38.806 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.805+0000 7f15ee7cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:28:38.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.805+0000 7f15ee7cf640 1 -- 172.21.15.3:0/2471921486 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15e8146790 con 0x7f15e80b4670 2024-03-25T18:28:38.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.834+0000 7f15e4ff9640 1 -- 172.21.15.3:0/2471921486 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f15d8052200 con 0x7f15e80b4670 2024-03-25T18:28:38.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.834+0000 7f15e4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:38.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.834+0000 7f15e4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:38.835 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:28:38.835 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:38.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.837+0000 7f15ee7cf640 10 librados: watch_flush enter 2024-03-25T18:28:38.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.837+0000 7f15ee7cf640 10 librados: watch_flush exit 2024-03-25T18:28:38.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.837+0000 7f15ee7cf640 20 client.4367.objecter shutdown clearing up homeless session... 2024-03-25T18:28:38.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.837+0000 7f15ee7cf640 10 client.4367.objecter successfully canceled tick 2024-03-25T18:28:38.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.837+0000 7f15ee7cf640 1 -- 172.21.15.3:0/2471921486 >> v1:172.21.15.3:6800/4195140966 conn(0x7f15d406dec0 legacy=0x7f15d4070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:38.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.837+0000 7f15ee7cf640 10 monclient: shutdown 2024-03-25T18:28:38.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.837+0000 7f15ee7cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:38.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.837+0000 7f15ee7cf640 1 -- 172.21.15.3:0/2471921486 >> v1:172.21.15.3:6790/0 conn(0x7f15e80b4670 legacy=0x7f15e813e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:38.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.837+0000 7f15ee7cf640 1 -- 172.21.15.3:0/2471921486 shutdown_connections 2024-03-25T18:28:38.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.837+0000 7f15ee7cf640 1 -- 172.21.15.3:0/2471921486 >> 172.21.15.3:0/2471921486 conn(0x7f15e80617a0 msgr2=0x7f15e81407f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:38.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.837+0000 7f15ee7cf640 1 -- 172.21.15.3:0/2471921486 shutdown_connections 2024-03-25T18:28:38.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.837+0000 7f15ee7cf640 1 -- 172.21.15.3:0/2471921486 wait complete. 2024-03-25T18:28:38.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:38.837+0000 7f15ee7cf640 1 librados: shutdown 2024-03-25T18:28:38.851 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:41.316 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:03:55.045 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:41.369 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:03:55.098 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:41.407 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:03:55.136 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:41.420 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:03:55.112 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:41.429 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:03:55.121 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:41.438 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:03:55.167 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:41.444 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:03:55.174 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:41.475 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:03:55.166 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:41.486 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:03:55.216 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:45.658 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:45.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.775+0000 7fb9b8c47640 10 monclient: get_monmap_and_config 2024-03-25T18:28:45.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.775+0000 7fb9b8c47640 10 monclient: build_initial_monmap 2024-03-25T18:28:45.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.775+0000 7fb9b8c47640 10 monclient: monmap: 2024-03-25T18:28:45.776 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:28:45.776 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:45.776 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:28:45.776137+0000 2024-03-25T18:28:45.776 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:28:45.776137+0000 2024-03-25T18:28:45.776 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:45.777 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:45.777 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:28:45.777 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:28:45.777 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:28:45.777 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:45.778 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.775+0000 7fb9b8c47640 1 Processor -- start 2024-03-25T18:28:45.778 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.775+0000 7fb9b8c47640 1 -- start start 2024-03-25T18:28:45.778 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.775+0000 7fb9b8c47640 10 monclient: init 2024-03-25T18:28:45.778 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.776+0000 7fb9b8c47640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:45.778 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.776+0000 7fb9b8c47640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:28:45.778 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.776+0000 7fb9b8c47640 10 monclient(hunting): picked mon.noname-a con 0x7fb9b40b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:28:45.778 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.776+0000 7fb9b8c47640 10 monclient(hunting): picked mon.noname-b con 0x7fb9b40b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:28:45.778 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.776+0000 7fb9b8c47640 10 monclient(hunting): picked mon.noname-c con 0x7fb9b413c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:28:45.778 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.776+0000 7fb9b8c47640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9b40ab0e0 con 0x7fb9b40b4ff0 2024-03-25T18:28:45.778 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.776+0000 7fb9b8c47640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9b40ab3c0 con 0x7fb9b413c480 2024-03-25T18:28:45.779 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.776+0000 7fb9b8c47640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9b40a9f30 con 0x7fb9b40b4670 2024-03-25T18:28:45.779 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.776+0000 7fb9b8c47640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:45.779 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.776+0000 7fb9b8c47640 10 monclient(hunting): authenticate will time out at 7580.037007s 2024-03-25T18:28:45.779 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.777+0000 7fb9b2d9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb9b413c480 0x7fb9b414dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:57262/0 (socket says 172.21.15.3:57262) 2024-03-25T18:28:45.779 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.777+0000 7fb9b2d9d640 1 -- 172.21.15.3:0/487308221 learned_addr learned my addr 172.21.15.3:0/487308221 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:28:45.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.844+0000 7fb9b1d9b640 1 -- 172.21.15.3:0/487308221 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4196900411 0 0) 0x7fb9b40ab0e0 con 0x7fb9b40b4ff0 2024-03-25T18:28:45.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.844+0000 7fb9b1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:28:45.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.844+0000 7fb9b1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:45.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.844+0000 7fb9b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:45.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.844+0000 7fb9b1d9b640 10 monclient(hunting): my global_id is 4401 2024-03-25T18:28:45.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.844+0000 7fb9b1d9b640 1 -- 172.21.15.3:0/487308221 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9a0003520 con 0x7fb9b40b4ff0 2024-03-25T18:28:45.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.849+0000 7fb9b1d9b640 1 -- 172.21.15.3:0/487308221 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2543665516 0 0) 0x7fb9a0003520 con 0x7fb9b40b4ff0 2024-03-25T18:28:45.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.849+0000 7fb9b1d9b640 1 -- 172.21.15.3:0/487308221 >> v1:172.21.15.3:6790/0 conn(0x7fb9b40b4670 legacy=0x7fb9b41425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:45.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.849+0000 7fb9b1d9b640 1 -- 172.21.15.3:0/487308221 >> v1:172.21.15.16:6789/0 conn(0x7fb9b413c480 legacy=0x7fb9b414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:45.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.849+0000 7fb9b1d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:28:45.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.849+0000 7fb9b1d9b640 1 monclient: found mon.noname-a 2024-03-25T18:28:45.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.849+0000 7fb9b1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:45.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.849+0000 7fb9b1d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:28:45.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.849+0000 7fb9b1d9b640 1 -- 172.21.15.3:0/487308221 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b40a9b70 con 0x7fb9b40b4ff0 2024-03-25T18:28:45.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.849+0000 7fb9b1d9b640 10 monclient: _finish_auth 0 2024-03-25T18:28:45.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.849+0000 7fb9b1d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:28:45.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.849+0000 7fb9b1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:45.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.849+0000 7fb9b1d9b640 1 -- 172.21.15.3:0/487308221 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fb9a4001690 con 0x7fb9b40b4ff0 2024-03-25T18:28:45.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.849+0000 7fb9b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:45.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.849+0000 7fb9b1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:45.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.849+0000 7fb9b1d9b640 10 monclient: dump: 2024-03-25T18:28:45.852 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:45.852 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:45.852 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:45.852 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:45.852 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:45.852 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:45.852 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:45.852 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:45.852 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:45.852 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:45.852 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.849+0000 7fb9b8c47640 5 monclient: authenticate success, global_id 4401 2024-03-25T18:28:45.852 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.849+0000 7fb9b8c47640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:45.863 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.862+0000 7fb9b1d9b640 1 -- 172.21.15.3:0/487308221 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9a40017f0 con 0x7fb9b40b4ff0 2024-03-25T18:28:45.863 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.862+0000 7fb9b1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:45.864 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.862+0000 7fb9b1d9b640 1 -- 172.21.15.3:0/487308221 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fb9a4001c40 con 0x7fb9b40b4ff0 2024-03-25T18:28:45.864 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.862+0000 7fb9b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:45.864 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.862+0000 7fb9b1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:45.864 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.862+0000 7fb9b1d9b640 10 monclient: dump: 2024-03-25T18:28:45.864 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:45.864 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:45.864 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:45.864 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:45.864 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:45.864 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:45.864 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:45.864 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:45.864 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:45.864 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:45.864 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.862+0000 7fb9b8c47640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:45.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.862+0000 7fb9b8c47640 4 set_mon_vals no callback set 2024-03-25T18:28:45.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.863+0000 7fb9b8c47640 10 monclient: shutdown 2024-03-25T18:28:45.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.863+0000 7fb9b8c47640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:45.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.863+0000 7fb9b8c47640 1 -- 172.21.15.3:0/487308221 >> v1:172.21.15.3:6789/0 conn(0x7fb9b40b4ff0 legacy=0x7fb9b413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:45.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.863+0000 7fb9b8c47640 1 -- 172.21.15.3:0/487308221 shutdown_connections 2024-03-25T18:28:45.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.863+0000 7fb9b8c47640 1 -- 172.21.15.3:0/487308221 >> 172.21.15.3:0/487308221 conn(0x7fb9b40617a0 msgr2=0x7fb9b4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:45.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.863+0000 7fb9b8c47640 1 -- 172.21.15.3:0/487308221 shutdown_connections 2024-03-25T18:28:45.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.863+0000 7fb9b8c47640 1 -- 172.21.15.3:0/487308221 wait complete. 2024-03-25T18:28:45.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.864+0000 7fb9b8c47640 10 monclient: build_initial_monmap 2024-03-25T18:28:45.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.864+0000 7fb9b8c47640 10 monclient: monmap: 2024-03-25T18:28:45.867 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:28:45.867 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:45.868 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:28:45.868 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:28:45.868 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:45.868 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:28:45.868 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:28:45.868 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:28:45.868 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:28:45.868 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:45.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.864+0000 7fb9b8c47640 1 librados: starting msgr at 2024-03-25T18:28:45.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.864+0000 7fb9b8c47640 1 librados: starting objecter 2024-03-25T18:28:45.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.865+0000 7fb9b8c47640 1 Processor -- start 2024-03-25T18:28:45.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.865+0000 7fb9b8c47640 1 -- start start 2024-03-25T18:28:45.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.865+0000 7fb9b8c47640 1 librados: setting wanted keys 2024-03-25T18:28:45.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.865+0000 7fb9b8c47640 1 librados: calling monclient init 2024-03-25T18:28:45.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.865+0000 7fb9b8c47640 10 monclient: init 2024-03-25T18:28:45.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.865+0000 7fb9b8c47640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:45.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.865+0000 7fb9b8c47640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:28:45.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.865+0000 7fb9b8c47640 10 monclient(hunting): picked mon.noname-c con 0x7fb9b40b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:28:45.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.865+0000 7fb9b8c47640 10 monclient(hunting): picked mon.noname-a con 0x7fb9b40b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:28:45.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.865+0000 7fb9b8c47640 10 monclient(hunting): picked mon.noname-b con 0x7fb9b413c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:28:45.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.865+0000 7fb9b8c47640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9b40ae630 con 0x7fb9b40b4ff0 2024-03-25T18:28:45.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.865+0000 7fb9b8c47640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9b40b1a00 con 0x7fb9b413c480 2024-03-25T18:28:45.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.865+0000 7fb9b8c47640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9b40b0880 con 0x7fb9b40b4670 2024-03-25T18:28:45.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.865+0000 7fb9b8c47640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:45.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.865+0000 7fb9b8c47640 10 monclient(hunting): authenticate will time out at 7580.126103s 2024-03-25T18:28:45.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.886+0000 7fb9a9d9b640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fb9b40b4ff0 0x7fb9b40aeac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:47794/0 (socket says 172.21.15.3:47794) 2024-03-25T18:28:45.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.886+0000 7fb9a9d9b640 1 -- 172.21.15.3:0/3776390684 learned_addr learned my addr 172.21.15.3:0/3776390684 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:28:45.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.887+0000 7fb9ab7fe640 10 client.?.objecter ms_handle_connect 0x7fb9b40b4ff0 2024-03-25T18:28:45.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.887+0000 7fb9ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:45.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.905+0000 7fb9ab7fe640 1 -- 172.21.15.3:0/3776390684 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899750663 0 0) 0x7fb9b40ae630 con 0x7fb9b40b4ff0 2024-03-25T18:28:45.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.905+0000 7fb9ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:45.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.905+0000 7fb9ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:45.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.905+0000 7fb9ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:45.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.905+0000 7fb9ab7fe640 10 monclient(hunting): my global_id is 4404 2024-03-25T18:28:45.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.905+0000 7fb9ab7fe640 1 -- 172.21.15.3:0/3776390684 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb994004eb0 con 0x7fb9b40b4ff0 2024-03-25T18:28:45.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.922+0000 7fb9ab7fe640 10 client.?.objecter ms_handle_connect 0x7fb9b40b4670 2024-03-25T18:28:45.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.922+0000 7fb9ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:45.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.923+0000 7fb9ab7fe640 10 client.?.objecter ms_handle_connect 0x7fb9b413c480 2024-03-25T18:28:45.924 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.923+0000 7fb9ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:45.941 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.940+0000 7fb9ab7fe640 1 -- 172.21.15.3:0/3776390684 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4005058667 0 0) 0x7fb9b40b1a00 con 0x7fb9b413c480 2024-03-25T18:28:45.941 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.940+0000 7fb9ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:45.941 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.940+0000 7fb9ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:45.941 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.940+0000 7fb9ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:45.941 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.940+0000 7fb9ab7fe640 10 monclient(hunting): my global_id is 4363 2024-03-25T18:28:45.942 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.940+0000 7fb9ab7fe640 1 -- 172.21.15.3:0/3776390684 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9b40ae630 con 0x7fb9b413c480 2024-03-25T18:28:45.942 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.940+0000 7fb9ab7fe640 1 -- 172.21.15.3:0/3776390684 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4086412086 0 0) 0x7fb994004eb0 con 0x7fb9b40b4ff0 2024-03-25T18:28:45.942 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.940+0000 7fb9ab7fe640 1 -- 172.21.15.3:0/3776390684 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9b40b1a00 con 0x7fb9b40b4ff0 2024-03-25T18:28:45.942 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.940+0000 7fb9ab7fe640 1 -- 172.21.15.3:0/3776390684 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fb99c0019d0 con 0x7fb9b40b4ff0 2024-03-25T18:28:45.942 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.940+0000 7fb9ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:45.942 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.940+0000 7fb9ab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:45.942 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.940+0000 7fb9ab7fe640 10 monclient(hunting): dump: 2024-03-25T18:28:45.942 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:45.942 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:45.942 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:45.942 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:45.942 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:45.942 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:45.942 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:45.943 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:45.943 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:45.943 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:45.943 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.940+0000 7fb9ab7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:45.944 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.943+0000 7fb9ab7fe640 1 -- 172.21.15.3:0/3776390684 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3102776532 0 0) 0x7fb9b40b0880 con 0x7fb9b40b4670 2024-03-25T18:28:45.944 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.943+0000 7fb9ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:45.944 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.943+0000 7fb9ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:45.944 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.943+0000 7fb9ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:45.944 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.943+0000 7fb9ab7fe640 10 monclient(hunting): my global_id is 4370 2024-03-25T18:28:45.945 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.943+0000 7fb9ab7fe640 1 -- 172.21.15.3:0/3776390684 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb994004eb0 con 0x7fb9b40b4670 2024-03-25T18:28:45.950 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.949+0000 7fb9ab7fe640 1 -- 172.21.15.3:0/3776390684 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4157546776 0 0) 0x7fb9b40ae630 con 0x7fb9b413c480 2024-03-25T18:28:45.950 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.949+0000 7fb9ab7fe640 1 -- 172.21.15.3:0/3776390684 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9b40b0880 con 0x7fb9b413c480 2024-03-25T18:28:45.950 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.949+0000 7fb9ab7fe640 1 -- 172.21.15.3:0/3776390684 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fb9ac001b00 con 0x7fb9b413c480 2024-03-25T18:28:45.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.949+0000 7fb9ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:45.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.949+0000 7fb9ab7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:45.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.949+0000 7fb9ab7fe640 10 monclient(hunting): dump: 2024-03-25T18:28:45.951 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:45.951 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:45.951 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:45.951 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:45.951 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:45.951 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:45.951 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:45.951 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:45.951 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:45.951 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:45.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.962+0000 7fb9ab7fe640 1 -- 172.21.15.3:0/3776390684 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1968912403 0 0) 0x7fb9b40b0880 con 0x7fb9b413c480 2024-03-25T18:28:45.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.962+0000 7fb9ab7fe640 1 -- 172.21.15.3:0/3776390684 >> v1:172.21.15.3:6790/0 conn(0x7fb9b40b4670 legacy=0x7fb9b40b12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:45.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.962+0000 7fb9ab7fe640 1 -- 172.21.15.3:0/3776390684 >> v1:172.21.15.3:6789/0 conn(0x7fb9b40b4ff0 legacy=0x7fb9b40aeac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:45.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.962+0000 7fb9ab7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:28:45.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.962+0000 7fb9ab7fe640 1 monclient: found mon.b 2024-03-25T18:28:45.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.962+0000 7fb9ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:45.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.962+0000 7fb9ab7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:28:45.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.962+0000 7fb9ab7fe640 1 -- 172.21.15.3:0/3776390684 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b414e1b0 con 0x7fb9b413c480 2024-03-25T18:28:45.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.962+0000 7fb9ab7fe640 10 monclient: _finish_auth 0 2024-03-25T18:28:45.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.962+0000 7fb9ab7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:28:45.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.962+0000 7fb9ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:45.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.962+0000 7fb9b8c47640 5 monclient: authenticate success, global_id 4363 2024-03-25T18:28:45.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.962+0000 7fb9b8c47640 10 monclient: _renew_subs 2024-03-25T18:28:45.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.962+0000 7fb9b8c47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:28:45.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.962+0000 7fb9b8c47640 1 -- 172.21.15.3:0/3776390684 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9b414e400 con 0x7fb9b413c480 2024-03-25T18:28:45.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.962+0000 7fb9b8c47640 10 client.4363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:45.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.962+0000 7fb9b8c47640 10 monclient: _renew_subs 2024-03-25T18:28:45.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.962+0000 7fb9b8c47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:28:45.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.962+0000 7fb9b8c47640 1 -- 172.21.15.3:0/3776390684 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9b41e3940 con 0x7fb9b413c480 2024-03-25T18:28:45.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.962+0000 7fb9b8c47640 1 librados: init done 2024-03-25T18:28:45.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.963+0000 7fb9b8c47640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:45.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.963+0000 7fb9b8c47640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:45.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.963+0000 7fb9b8c47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:28:45.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:45.963+0000 7fb9b8c47640 1 -- 172.21.15.3:0/3776390684 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9b406dbd0 con 0x7fb9b413c480 2024-03-25T18:28:46.275 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.273+0000 7fb9ab7fe640 1 -- 172.21.15.3:0/3776390684 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9ac001f10 con 0x7fb9b413c480 2024-03-25T18:28:46.275 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.273+0000 7fb9ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:46.275 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.273+0000 7fb9ab7fe640 1 -- 172.21.15.3:0/3776390684 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fb9ac004d00 con 0x7fb9b413c480 2024-03-25T18:28:46.275 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.273+0000 7fb9ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:46.275 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.273+0000 7fb9ab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:46.275 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.273+0000 7fb9ab7fe640 10 monclient: dump: 2024-03-25T18:28:46.275 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:46.276 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:46.276 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:46.276 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:46.276 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:46.276 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:46.276 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:46.276 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:46.276 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:46.276 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:46.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.273+0000 7fb9b159a640 4 set_mon_vals no callback set 2024-03-25T18:28:46.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.274+0000 7fb9ab7fe640 1 -- 172.21.15.3:0/3776390684 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7fb9ac004f20 con 0x7fb9b413c480 2024-03-25T18:28:46.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.275+0000 7fb9b259c640 10 monclient: get_auth_request con 0x7fb99406df70 auth_method 0 2024-03-25T18:28:46.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.275+0000 7fb9ab7fe640 10 client.4363.objecter ms_handle_connect 0x7fb99406df70 2024-03-25T18:28:46.279 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.277+0000 7fb9ab7fe640 1 -- 172.21.15.3:0/3776390684 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7fb9ac054920 con 0x7fb9b413c480 2024-03-25T18:28:46.279 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.277+0000 7fb9ab7fe640 10 client.4363.objecter ms_dispatch 0x7fb9b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:46.279 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.277+0000 7fb9ab7fe640 3 client.4363.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:46.279 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.277+0000 7fb9ab7fe640 3 client.4363.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:46.279 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.278+0000 7fb9ab7fe640 20 client.4363.objecter dump_active .. 0 homeless 2024-03-25T18:28:46.281 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.280+0000 7fb9ab7fe640 1 -- 172.21.15.3:0/3776390684 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb9ac050430 con 0x7fb9b413c480 2024-03-25T18:28:46.281 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.280+0000 7fb9ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:46.281 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.280+0000 7fb9ab7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:46.529 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.528+0000 7fb9b8c47640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:46.530 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.528+0000 7fb9b8c47640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:46.530 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.528+0000 7fb9b8c47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:28:46.530 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.528+0000 7fb9b8c47640 1 -- 172.21.15.3:0/3776390684 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9b4146790 con 0x7fb9b413c480 2024-03-25T18:28:46.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.568+0000 7fb9ab7fe640 1 -- 172.21.15.3:0/3776390684 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7fb9ac052360 con 0x7fb9b413c480 2024-03-25T18:28:46.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.568+0000 7fb9ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:46.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.568+0000 7fb9ab7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:46.570 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:28:46.570 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:46.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.571+0000 7fb9b8c47640 10 librados: watch_flush enter 2024-03-25T18:28:46.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.571+0000 7fb9b8c47640 10 librados: watch_flush exit 2024-03-25T18:28:46.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.571+0000 7fb9b8c47640 20 client.4363.objecter shutdown clearing up homeless session... 2024-03-25T18:28:46.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.571+0000 7fb9b8c47640 10 client.4363.objecter successfully canceled tick 2024-03-25T18:28:46.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.571+0000 7fb9b8c47640 1 -- 172.21.15.3:0/3776390684 >> v1:172.21.15.3:6800/4195140966 conn(0x7fb99406df70 legacy=0x7fb994070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:46.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.571+0000 7fb9b8c47640 10 monclient: shutdown 2024-03-25T18:28:46.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.571+0000 7fb9b8c47640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:46.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.571+0000 7fb9b8c47640 1 -- 172.21.15.3:0/3776390684 >> v1:172.21.15.16:6789/0 conn(0x7fb9b413c480 legacy=0x7fb9b40af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:46.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.571+0000 7fb9b8c47640 1 -- 172.21.15.3:0/3776390684 shutdown_connections 2024-03-25T18:28:46.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.571+0000 7fb9b8c47640 1 -- 172.21.15.3:0/3776390684 >> 172.21.15.3:0/3776390684 conn(0x7fb9b40617a0 msgr2=0x7fb9b41407a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:46.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.571+0000 7fb9b8c47640 1 -- 172.21.15.3:0/3776390684 shutdown_connections 2024-03-25T18:28:46.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.571+0000 7fb9b8c47640 1 -- 172.21.15.3:0/3776390684 wait complete. 2024-03-25T18:28:46.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:46.571+0000 7fb9b8c47640 1 librados: shutdown 2024-03-25T18:28:46.584 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:53.392 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:53.509 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.507+0000 7f17b1074640 10 monclient: get_monmap_and_config 2024-03-25T18:28:53.509 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.507+0000 7f17b1074640 10 monclient: build_initial_monmap 2024-03-25T18:28:53.509 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.507+0000 7f17b1074640 10 monclient: monmap: 2024-03-25T18:28:53.509 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:28:53.509 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:53.509 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:28:53.508868+0000 2024-03-25T18:28:53.509 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:28:53.508868+0000 2024-03-25T18:28:53.509 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:53.509 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:53.509 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:28:53.510 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:28:53.510 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:28:53.510 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:53.511 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.508+0000 7f17b1074640 1 Processor -- start 2024-03-25T18:28:53.511 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.508+0000 7f17b1074640 1 -- start start 2024-03-25T18:28:53.511 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.508+0000 7f17b1074640 10 monclient: init 2024-03-25T18:28:53.511 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.509+0000 7f17b1074640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:53.511 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.509+0000 7f17b1074640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:28:53.511 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.509+0000 7f17b1074640 10 monclient(hunting): picked mon.noname-a con 0x7f17ac0b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:28:53.511 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.509+0000 7f17b1074640 10 monclient(hunting): picked mon.noname-c con 0x7f17ac0b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:28:53.511 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.509+0000 7f17b1074640 10 monclient(hunting): picked mon.noname-b con 0x7f17ac13c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:28:53.511 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.509+0000 7f17b1074640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17ac0ab0e0 con 0x7f17ac0b4ff0 2024-03-25T18:28:53.511 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.509+0000 7f17b1074640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17ac0ab3c0 con 0x7f17ac0b4670 2024-03-25T18:28:53.512 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.509+0000 7f17b1074640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17ac0a9f30 con 0x7f17ac13c480 2024-03-25T18:28:53.512 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.509+0000 7f17b1074640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:53.512 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.509+0000 7f17b1074640 10 monclient(hunting): authenticate will time out at 7587.769790s 2024-03-25T18:28:53.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.520+0000 7f17aa59c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f17ac0b4670 0x7f17ac1425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:56360/0 (socket says 172.21.15.3:56360) 2024-03-25T18:28:53.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.520+0000 7f17aa59c640 1 -- 172.21.15.3:0/1196411654 learned_addr learned my addr 172.21.15.3:0/1196411654 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:28:53.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.534+0000 7f17a9d9b640 1 -- 172.21.15.3:0/1196411654 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3535697882 0 0) 0x7f17ac0ab3c0 con 0x7f17ac0b4670 2024-03-25T18:28:53.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.534+0000 7f17a9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:28:53.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.534+0000 7f17a9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:53.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.534+0000 7f17a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:53.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.534+0000 7f17a9d9b640 10 monclient(hunting): my global_id is 4366 2024-03-25T18:28:53.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.534+0000 7f17a9d9b640 1 -- 172.21.15.3:0/1196411654 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1794003520 con 0x7f17ac0b4670 2024-03-25T18:28:53.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.539+0000 7f17a9d9b640 1 -- 172.21.15.3:0/1196411654 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2508571835 0 0) 0x7f17ac0a9f30 con 0x7f17ac13c480 2024-03-25T18:28:53.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.539+0000 7f17a9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:28:53.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.539+0000 7f17a9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:53.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.539+0000 7f17a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:53.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.539+0000 7f17a9d9b640 10 monclient(hunting): my global_id is 4373 2024-03-25T18:28:53.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.539+0000 7f17a9d9b640 1 -- 172.21.15.3:0/1196411654 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17ac0ab3c0 con 0x7f17ac13c480 2024-03-25T18:28:53.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.557+0000 7f17a9d9b640 1 -- 172.21.15.3:0/1196411654 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1824649576 0 0) 0x7f17ac0ab0e0 con 0x7f17ac0b4ff0 2024-03-25T18:28:53.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.557+0000 7f17a9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:28:53.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.557+0000 7f17a9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:53.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.557+0000 7f17a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:53.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.557+0000 7f17a9d9b640 10 monclient(hunting): my global_id is 4407 2024-03-25T18:28:53.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.557+0000 7f17a9d9b640 1 -- 172.21.15.3:0/1196411654 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17ac0a9f30 con 0x7f17ac0b4ff0 2024-03-25T18:28:53.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.561+0000 7f17a9d9b640 1 -- 172.21.15.3:0/1196411654 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 729060120 0 0) 0x7f1794003520 con 0x7f17ac0b4670 2024-03-25T18:28:53.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.561+0000 7f17a9d9b640 1 -- 172.21.15.3:0/1196411654 >> v1:172.21.15.3:6790/0 conn(0x7f17ac13c480 legacy=0x7f17ac14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:53.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.561+0000 7f17a9d9b640 1 -- 172.21.15.3:0/1196411654 >> v1:172.21.15.3:6789/0 conn(0x7f17ac0b4ff0 legacy=0x7f17ac13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:53.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.561+0000 7f17a9d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:28:53.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.561+0000 7f17a9d9b640 1 monclient: found mon.noname-c 2024-03-25T18:28:53.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.561+0000 7f17a9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:53.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.561+0000 7f17a9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:28:53.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.561+0000 7f17a9d9b640 1 -- 172.21.15.3:0/1196411654 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ac0a9b70 con 0x7f17ac0b4670 2024-03-25T18:28:53.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.561+0000 7f17a9d9b640 10 monclient: _finish_auth 0 2024-03-25T18:28:53.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.561+0000 7f17a9d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:28:53.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.561+0000 7f17a9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:53.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.561+0000 7f17a9d9b640 1 -- 172.21.15.3:0/1196411654 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f17a0001690 con 0x7f17ac0b4670 2024-03-25T18:28:53.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.561+0000 7f17b1074640 5 monclient: authenticate success, global_id 4366 2024-03-25T18:28:53.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.561+0000 7f17b1074640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:53.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.561+0000 7f17a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:53.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.561+0000 7f17a9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:53.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.561+0000 7f17a9d9b640 10 monclient: dump: 2024-03-25T18:28:53.564 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:53.564 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:53.564 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:53.564 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:53.564 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:53.564 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:53.564 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:53.564 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:53.564 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:53.564 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:53.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.561+0000 7f17b1074640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:53.593 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.592+0000 7f17a9d9b640 1 -- 172.21.15.3:0/1196411654 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17a00017f0 con 0x7f17ac0b4670 2024-03-25T18:28:53.593 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.592+0000 7f17a9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:53.593 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.592+0000 7f17a9d9b640 1 -- 172.21.15.3:0/1196411654 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f17a0001c00 con 0x7f17ac0b4670 2024-03-25T18:28:53.593 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.592+0000 7f17a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:53.594 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.592+0000 7f17a9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:53.594 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.592+0000 7f17a9d9b640 10 monclient: dump: 2024-03-25T18:28:53.594 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:53.594 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:53.594 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:53.594 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:53.594 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:53.594 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:53.594 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:53.594 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:53.594 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:53.594 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:53.594 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.592+0000 7f17b1074640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:53.595 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.592+0000 7f17b1074640 4 set_mon_vals no callback set 2024-03-25T18:28:53.596 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.592+0000 7f17b1074640 10 monclient: shutdown 2024-03-25T18:28:53.596 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.592+0000 7f17b1074640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:53.597 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.592+0000 7f17b1074640 1 -- 172.21.15.3:0/1196411654 >> v1:172.21.15.16:6789/0 conn(0x7f17ac0b4670 legacy=0x7f17ac1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:53.597 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.593+0000 7f17b1074640 1 -- 172.21.15.3:0/1196411654 shutdown_connections 2024-03-25T18:28:53.597 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.593+0000 7f17b1074640 1 -- 172.21.15.3:0/1196411654 >> 172.21.15.3:0/1196411654 conn(0x7f17ac0617a0 msgr2=0x7f17ac061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:53.597 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.593+0000 7f17b1074640 1 -- 172.21.15.3:0/1196411654 shutdown_connections 2024-03-25T18:28:53.597 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.593+0000 7f17b1074640 1 -- 172.21.15.3:0/1196411654 wait complete. 2024-03-25T18:28:53.597 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.593+0000 7f17b1074640 10 monclient: build_initial_monmap 2024-03-25T18:28:53.597 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.593+0000 7f17b1074640 10 monclient: monmap: 2024-03-25T18:28:53.597 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:28:53.597 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:53.597 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:28:53.597 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:28:53.597 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:53.597 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:28:53.597 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:28:53.598 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:28:53.598 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:28:53.598 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:53.598 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.594+0000 7f17b1074640 1 librados: starting msgr at 2024-03-25T18:28:53.598 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.594+0000 7f17b1074640 1 librados: starting objecter 2024-03-25T18:28:53.598 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.594+0000 7f17b1074640 1 Processor -- start 2024-03-25T18:28:53.598 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.594+0000 7f17b1074640 1 -- start start 2024-03-25T18:28:53.598 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.594+0000 7f17b1074640 1 librados: setting wanted keys 2024-03-25T18:28:53.598 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.594+0000 7f17b1074640 1 librados: calling monclient init 2024-03-25T18:28:53.598 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.594+0000 7f17b1074640 10 monclient: init 2024-03-25T18:28:53.598 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.595+0000 7f17b1074640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:53.598 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.595+0000 7f17b1074640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:53.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.595+0000 7f17b1074640 10 monclient(hunting): picked mon.noname-b con 0x7f17ac0b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:28:53.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.595+0000 7f17b1074640 10 monclient(hunting): picked mon.noname-a con 0x7f17ac0b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:28:53.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.595+0000 7f17b1074640 10 monclient(hunting): picked mon.noname-c con 0x7f17ac13c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:28:53.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.595+0000 7f17b1074640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17ac143810 con 0x7f17ac0b4ff0 2024-03-25T18:28:53.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.595+0000 7f17b1074640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17ac130ad0 con 0x7f17ac0b4670 2024-03-25T18:28:53.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.595+0000 7f17b1074640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17ac144120 con 0x7f17ac13c480 2024-03-25T18:28:53.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.595+0000 7f17b1074640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:53.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.595+0000 7f17b1074640 10 monclient(hunting): authenticate will time out at 7587.855779s 2024-03-25T18:28:53.604 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.603+0000 7f17aa59c640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f17ac0b4ff0 0x7f17ac12fcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:46694/0 (socket says 172.21.15.3:46694) 2024-03-25T18:28:53.604 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.603+0000 7f17aa59c640 1 -- 172.21.15.3:0/2215659063 learned_addr learned my addr 172.21.15.3:0/2215659063 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:28:53.605 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.604+0000 7f17937fe640 10 client.?.objecter ms_handle_connect 0x7f17ac0b4ff0 2024-03-25T18:28:53.605 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.604+0000 7f17937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:53.609 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.608+0000 7f17937fe640 10 client.?.objecter ms_handle_connect 0x7f17ac0b4670 2024-03-25T18:28:53.609 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.608+0000 7f17937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:53.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.616+0000 7f17937fe640 1 -- 172.21.15.3:0/2215659063 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2247747896 0 0) 0x7f17ac143810 con 0x7f17ac0b4ff0 2024-03-25T18:28:53.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.616+0000 7f17937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:53.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.616+0000 7f17937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:53.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.616+0000 7f17937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:53.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.616+0000 7f17937fe640 10 monclient(hunting): my global_id is 4410 2024-03-25T18:28:53.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.616+0000 7f17937fe640 1 -- 172.21.15.3:0/2215659063 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1798004eb0 con 0x7f17ac0b4ff0 2024-03-25T18:28:53.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.617+0000 7f17937fe640 10 client.?.objecter ms_handle_connect 0x7f17ac13c480 2024-03-25T18:28:53.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.617+0000 7f17937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:53.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.629+0000 7f17937fe640 1 -- 172.21.15.3:0/2215659063 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 867327916 0 0) 0x7f17ac130ad0 con 0x7f17ac0b4670 2024-03-25T18:28:53.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.629+0000 7f17937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:53.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.629+0000 7f17937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:53.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.629+0000 7f17937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:53.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.629+0000 7f17937fe640 10 monclient(hunting): my global_id is 4369 2024-03-25T18:28:53.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.629+0000 7f17937fe640 1 -- 172.21.15.3:0/2215659063 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17ac143810 con 0x7f17ac0b4670 2024-03-25T18:28:53.637 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.636+0000 7f17937fe640 1 -- 172.21.15.3:0/2215659063 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 221070683 0 0) 0x7f17ac144120 con 0x7f17ac13c480 2024-03-25T18:28:53.637 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.636+0000 7f17937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:53.637 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.636+0000 7f17937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:53.637 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.636+0000 7f17937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:53.637 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.636+0000 7f17937fe640 10 monclient(hunting): my global_id is 4376 2024-03-25T18:28:53.638 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.636+0000 7f17937fe640 1 -- 172.21.15.3:0/2215659063 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17ac130ad0 con 0x7f17ac13c480 2024-03-25T18:28:53.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.649+0000 7f17937fe640 1 -- 172.21.15.3:0/2215659063 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 550566516 0 0) 0x7f17ac143810 con 0x7f17ac0b4670 2024-03-25T18:28:53.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.649+0000 7f17937fe640 1 -- 172.21.15.3:0/2215659063 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17ac144120 con 0x7f17ac0b4670 2024-03-25T18:28:53.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.649+0000 7f17937fe640 1 -- 172.21.15.3:0/2215659063 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f179c0018f0 con 0x7f17ac0b4670 2024-03-25T18:28:53.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.649+0000 7f17937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:53.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.649+0000 7f17937fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:53.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.649+0000 7f17937fe640 10 monclient(hunting): dump: 2024-03-25T18:28:53.651 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:53.651 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:53.651 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:53.651 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:53.651 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:53.651 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:53.651 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:53.651 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:53.651 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:53.651 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:53.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.649+0000 7f17937fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:53.657 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.656+0000 7f17937fe640 1 -- 172.21.15.3:0/2215659063 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3134882919 0 0) 0x7f1798004eb0 con 0x7f17ac0b4ff0 2024-03-25T18:28:53.657 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.656+0000 7f17937fe640 1 -- 172.21.15.3:0/2215659063 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17ac143810 con 0x7f17ac0b4ff0 2024-03-25T18:28:53.657 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.656+0000 7f17937fe640 1 -- 172.21.15.3:0/2215659063 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f17a0001bc0 con 0x7f17ac0b4ff0 2024-03-25T18:28:53.657 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.656+0000 7f17937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:53.657 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.656+0000 7f17937fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:53.657 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.656+0000 7f17937fe640 10 monclient(hunting): dump: 2024-03-25T18:28:53.658 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:53.658 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:53.658 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:53.658 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:53.658 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:53.658 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:53.658 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:53.658 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:53.658 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:53.658 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:53.661 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.659+0000 7f17937fe640 1 -- 172.21.15.3:0/2215659063 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1181099098 0 0) 0x7f17ac130ad0 con 0x7f17ac13c480 2024-03-25T18:28:53.661 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.660+0000 7f17937fe640 1 -- 172.21.15.3:0/2215659063 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1798004eb0 con 0x7f17ac13c480 2024-03-25T18:28:53.661 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.660+0000 7f17937fe640 1 -- 172.21.15.3:0/2215659063 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f17a4001a70 con 0x7f17ac13c480 2024-03-25T18:28:53.661 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.660+0000 7f17937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:53.661 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.660+0000 7f17937fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:53.661 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.660+0000 7f17937fe640 10 monclient(hunting): dump: 2024-03-25T18:28:53.661 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:53.661 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:53.661 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:53.661 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:53.661 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:53.661 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:53.661 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:53.662 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:53.662 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:53.662 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:53.668 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.667+0000 7f17937fe640 1 -- 172.21.15.3:0/2215659063 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 364691380 0 0) 0x7f17ac144120 con 0x7f17ac0b4670 2024-03-25T18:28:53.668 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.667+0000 7f17937fe640 1 -- 172.21.15.3:0/2215659063 >> v1:172.21.15.3:6790/0 conn(0x7f17ac13c480 legacy=0x7f17ac1303c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:53.668 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.667+0000 7f17937fe640 1 -- 172.21.15.3:0/2215659063 >> v1:172.21.15.3:6789/0 conn(0x7f17ac0b4ff0 legacy=0x7f17ac12fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:53.668 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.667+0000 7f17937fe640 10 monclient: _finish_hunting 0 2024-03-25T18:28:53.668 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.667+0000 7f17937fe640 1 monclient: found mon.b 2024-03-25T18:28:53.669 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.667+0000 7f17937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:53.669 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.667+0000 7f17937fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:28:53.669 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.667+0000 7f17937fe640 1 -- 172.21.15.3:0/2215659063 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ac144400 con 0x7f17ac0b4670 2024-03-25T18:28:53.669 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.667+0000 7f17937fe640 10 monclient: _finish_auth 0 2024-03-25T18:28:53.669 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.667+0000 7f17937fe640 10 monclient: _check_auth_tickets 2024-03-25T18:28:53.669 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.667+0000 7f17937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:53.669 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.667+0000 7f17b1074640 5 monclient: authenticate success, global_id 4369 2024-03-25T18:28:53.669 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.667+0000 7f17b1074640 10 monclient: _renew_subs 2024-03-25T18:28:53.669 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.667+0000 7f17b1074640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:28:53.669 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.667+0000 7f17b1074640 1 -- 172.21.15.3:0/2215659063 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17ac1446e0 con 0x7f17ac0b4670 2024-03-25T18:28:53.669 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.667+0000 7f17b1074640 10 client.4369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:53.669 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.667+0000 7f17b1074640 10 monclient: _renew_subs 2024-03-25T18:28:53.669 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.667+0000 7f17b1074640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:28:53.669 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.667+0000 7f17b1074640 1 -- 172.21.15.3:0/2215659063 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f17ac1454f0 con 0x7f17ac0b4670 2024-03-25T18:28:53.670 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.667+0000 7f17b1074640 1 librados: init done 2024-03-25T18:28:53.670 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.668+0000 7f17b1074640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:53.670 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.668+0000 7f17b1074640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:53.670 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.668+0000 7f17b1074640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:28:53.670 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.668+0000 7f17b1074640 1 -- 172.21.15.3:0/2215659063 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17ac06dbd0 con 0x7f17ac0b4670 2024-03-25T18:28:53.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.953+0000 7f17937fe640 1 -- 172.21.15.3:0/2215659063 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f179c003610 con 0x7f17ac0b4670 2024-03-25T18:28:53.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.953+0000 7f17937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:53.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.953+0000 7f17937fe640 1 -- 172.21.15.3:0/2215659063 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f179c003920 con 0x7f17ac0b4670 2024-03-25T18:28:53.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.953+0000 7f17937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:53.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.953+0000 7f17937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:53.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.953+0000 7f17937fe640 10 monclient: dump: 2024-03-25T18:28:53.955 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:28:53.955 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:28:53.955 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:53.955 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:28:53.955 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:53.955 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:28:53.955 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:28:53.956 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:28:53.956 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:28:53.956 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:28:53.956 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.953+0000 7f17a959a640 4 set_mon_vals no callback set 2024-03-25T18:28:53.956 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.954+0000 7f17937fe640 1 -- 172.21.15.3:0/2215659063 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f179c0198f0 con 0x7f17ac0b4670 2024-03-25T18:28:53.956 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.954+0000 7f17aa59c640 10 monclient: get_auth_request con 0x7f179806e0b0 auth_method 0 2024-03-25T18:28:53.956 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.954+0000 7f17937fe640 10 client.4369.objecter ms_handle_connect 0x7f179806e0b0 2024-03-25T18:28:53.962 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.959+0000 7f17937fe640 1 -- 172.21.15.3:0/2215659063 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f179c053610 con 0x7f17ac0b4670 2024-03-25T18:28:53.962 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.959+0000 7f17937fe640 10 client.4369.objecter ms_dispatch 0x7f17ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:53.962 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.959+0000 7f17937fe640 3 client.4369.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:53.962 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.959+0000 7f17937fe640 3 client.4369.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:53.962 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.959+0000 7f17937fe640 20 client.4369.objecter dump_active .. 0 homeless 2024-03-25T18:28:53.962 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.959+0000 7f17937fe640 1 -- 172.21.15.3:0/2215659063 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f179c04ef50 con 0x7f17ac0b4670 2024-03-25T18:28:53.962 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.959+0000 7f17937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:53.962 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:53.959+0000 7f17937fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:54.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:54.217+0000 7f17b1074640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:54.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:54.217+0000 7f17b1074640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:54.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:54.217+0000 7f17b1074640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:28:54.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:54.217+0000 7f17b1074640 1 -- 172.21.15.3:0/2215659063 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17ac0688c0 con 0x7f17ac0b4670 2024-03-25T18:28:54.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:54.250+0000 7f17937fe640 1 -- 172.21.15.3:0/2215659063 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f179c051050 con 0x7f17ac0b4670 2024-03-25T18:28:54.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:54.250+0000 7f17937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:54.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:54.250+0000 7f17937fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:54.252 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:28:54.252 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:54.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:54.253+0000 7f17b1074640 10 librados: watch_flush enter 2024-03-25T18:28:54.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:54.253+0000 7f17b1074640 10 librados: watch_flush exit 2024-03-25T18:28:54.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:54.253+0000 7f17b1074640 20 client.4369.objecter shutdown clearing up homeless session... 2024-03-25T18:28:54.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:54.253+0000 7f17b1074640 10 client.4369.objecter successfully canceled tick 2024-03-25T18:28:54.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:54.253+0000 7f17b1074640 1 -- 172.21.15.3:0/2215659063 >> v1:172.21.15.3:6800/4195140966 conn(0x7f179806e0b0 legacy=0x7f1798070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:54.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:54.253+0000 7f17b1074640 10 monclient: shutdown 2024-03-25T18:28:54.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:54.253+0000 7f17b1074640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:54.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:54.253+0000 7f17b1074640 1 -- 172.21.15.3:0/2215659063 >> v1:172.21.15.16:6789/0 conn(0x7f17ac0b4670 legacy=0x7f17ac144de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:54.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:54.253+0000 7f17b1074640 1 -- 172.21.15.3:0/2215659063 shutdown_connections 2024-03-25T18:28:54.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:54.254+0000 7f17b1074640 1 -- 172.21.15.3:0/2215659063 >> 172.21.15.3:0/2215659063 conn(0x7f17ac0617a0 msgr2=0x7f17ac140760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:54.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:54.254+0000 7f17b1074640 1 -- 172.21.15.3:0/2215659063 shutdown_connections 2024-03-25T18:28:54.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:54.254+0000 7f17b1074640 1 -- 172.21.15.3:0/2215659063 wait complete. 2024-03-25T18:28:54.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:28:54.254+0000 7f17b1074640 1 librados: shutdown 2024-03-25T18:28:54.267 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:01.074 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:01.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.192+0000 7fba50359640 10 monclient: get_monmap_and_config 2024-03-25T18:29:01.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.192+0000 7fba50359640 10 monclient: build_initial_monmap 2024-03-25T18:29:01.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.192+0000 7fba50359640 10 monclient: monmap: 2024-03-25T18:29:01.194 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:29:01.194 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:01.194 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:29:01.193919+0000 2024-03-25T18:29:01.194 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:29:01.193919+0000 2024-03-25T18:29:01.194 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:01.194 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:01.194 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:29:01.194 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:29:01.194 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:29:01.195 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:01.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.193+0000 7fba50359640 1 Processor -- start 2024-03-25T18:29:01.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.193+0000 7fba50359640 1 -- start start 2024-03-25T18:29:01.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.193+0000 7fba50359640 10 monclient: init 2024-03-25T18:29:01.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.194+0000 7fba50359640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:01.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.194+0000 7fba50359640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:29:01.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.194+0000 7fba50359640 10 monclient(hunting): picked mon.noname-b con 0x7fba480615c0 addr v1:172.21.15.3:6790/0 2024-03-25T18:29:01.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.194+0000 7fba50359640 10 monclient(hunting): picked mon.noname-a con 0x7fba480619a0 addr v1:172.21.15.3:6789/0 2024-03-25T18:29:01.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.194+0000 7fba50359640 10 monclient(hunting): picked mon.noname-c con 0x7fba48143b80 addr v1:172.21.15.16:6789/0 2024-03-25T18:29:01.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.194+0000 7fba50359640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba48072d90 con 0x7fba480619a0 2024-03-25T18:29:01.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.194+0000 7fba50359640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba480ae410 con 0x7fba48143b80 2024-03-25T18:29:01.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.194+0000 7fba50359640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba480727d0 con 0x7fba480615c0 2024-03-25T18:29:01.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.194+0000 7fba50359640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:01.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.194+0000 7fba50359640 10 monclient(hunting): authenticate will time out at 7595.454931s 2024-03-25T18:29:01.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.196+0000 7fba4d8f4640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fba480619a0 0x7fba48140140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:49572/0 (socket says 172.21.15.3:49572) 2024-03-25T18:29:01.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.196+0000 7fba4d8f4640 1 -- 172.21.15.3:0/2068621181 learned_addr learned my addr 172.21.15.3:0/2068621181 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:29:01.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.209+0000 7fba4d0f3640 1 -- 172.21.15.3:0/2068621181 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3949241775 0 0) 0x7fba48072d90 con 0x7fba480619a0 2024-03-25T18:29:01.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.209+0000 7fba4d0f3640 10 monclient(hunting): _negotiate 2024-03-25T18:29:01.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.209+0000 7fba4d0f3640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:01.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.209+0000 7fba4d0f3640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:01.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.209+0000 7fba4d0f3640 10 monclient(hunting): my global_id is 4413 2024-03-25T18:29:01.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.209+0000 7fba4d0f3640 1 -- 172.21.15.3:0/2068621181 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba34003520 con 0x7fba480619a0 2024-03-25T18:29:01.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.212+0000 7fba4d0f3640 1 -- 172.21.15.3:0/2068621181 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2923765251 0 0) 0x7fba480ae410 con 0x7fba48143b80 2024-03-25T18:29:01.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.212+0000 7fba4d0f3640 10 monclient(hunting): _negotiate 2024-03-25T18:29:01.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.212+0000 7fba4d0f3640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:01.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.212+0000 7fba4d0f3640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:01.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.212+0000 7fba4d0f3640 10 monclient(hunting): my global_id is 4372 2024-03-25T18:29:01.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.212+0000 7fba4d0f3640 1 -- 172.21.15.3:0/2068621181 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba48072d90 con 0x7fba48143b80 2024-03-25T18:29:01.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.219+0000 7fba4d0f3640 1 -- 172.21.15.3:0/2068621181 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1770751885 0 0) 0x7fba34003520 con 0x7fba480619a0 2024-03-25T18:29:01.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.219+0000 7fba4d0f3640 1 -- 172.21.15.3:0/2068621181 >> v1:172.21.15.3:6790/0 conn(0x7fba480615c0 legacy=0x7fba4813d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:01.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.219+0000 7fba4d0f3640 1 -- 172.21.15.3:0/2068621181 >> v1:172.21.15.16:6789/0 conn(0x7fba48143b80 legacy=0x7fba4814ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:01.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.219+0000 7fba4d0f3640 10 monclient: _finish_hunting 0 2024-03-25T18:29:01.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.219+0000 7fba4d0f3640 1 monclient: found mon.noname-a 2024-03-25T18:29:01.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.219+0000 7fba4d0f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:01.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.219+0000 7fba4d0f3640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:29:01.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.219+0000 7fba4d0f3640 1 -- 172.21.15.3:0/2068621181 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba48072ab0 con 0x7fba480619a0 2024-03-25T18:29:01.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.219+0000 7fba4d0f3640 10 monclient: _finish_auth 0 2024-03-25T18:29:01.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.219+0000 7fba4d0f3640 10 monclient: _check_auth_tickets 2024-03-25T18:29:01.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.219+0000 7fba4d0f3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:01.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.219+0000 7fba4d0f3640 1 -- 172.21.15.3:0/2068621181 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fba38001690 con 0x7fba480619a0 2024-03-25T18:29:01.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.219+0000 7fba50359640 5 monclient: authenticate success, global_id 4413 2024-03-25T18:29:01.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.219+0000 7fba50359640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:01.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.219+0000 7fba4d0f3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:01.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.219+0000 7fba4d0f3640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:01.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.219+0000 7fba4d0f3640 10 monclient: dump: 2024-03-25T18:29:01.223 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:01.223 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:01.223 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:01.223 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:01.223 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:01.223 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:01.223 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:01.223 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:01.223 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:01.223 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:01.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.219+0000 7fba50359640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:01.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.239+0000 7fba4d0f3640 1 -- 172.21.15.3:0/2068621181 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba380017f0 con 0x7fba480619a0 2024-03-25T18:29:01.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.239+0000 7fba4d0f3640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:01.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.239+0000 7fba4d0f3640 1 -- 172.21.15.3:0/2068621181 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fba38001b80 con 0x7fba480619a0 2024-03-25T18:29:01.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.239+0000 7fba4d0f3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:01.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.239+0000 7fba4d0f3640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:01.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.239+0000 7fba4d0f3640 10 monclient: dump: 2024-03-25T18:29:01.242 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:01.242 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:01.242 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:01.242 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:01.243 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:01.243 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:01.243 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:01.243 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:01.243 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:01.243 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:01.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.239+0000 7fba50359640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:01.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.239+0000 7fba50359640 4 set_mon_vals no callback set 2024-03-25T18:29:01.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.239+0000 7fba50359640 10 monclient: shutdown 2024-03-25T18:29:01.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.239+0000 7fba50359640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:01.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.239+0000 7fba50359640 1 -- 172.21.15.3:0/2068621181 >> v1:172.21.15.3:6789/0 conn(0x7fba480619a0 legacy=0x7fba48140140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:01.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.240+0000 7fba50359640 1 -- 172.21.15.3:0/2068621181 shutdown_connections 2024-03-25T18:29:01.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.240+0000 7fba50359640 1 -- 172.21.15.3:0/2068621181 >> 172.21.15.3:0/2068621181 conn(0x7fba480b1270 msgr2=0x7fba480b1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:01.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.240+0000 7fba50359640 1 -- 172.21.15.3:0/2068621181 shutdown_connections 2024-03-25T18:29:01.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.240+0000 7fba50359640 1 -- 172.21.15.3:0/2068621181 wait complete. 2024-03-25T18:29:01.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.241+0000 7fba50359640 10 monclient: build_initial_monmap 2024-03-25T18:29:01.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.241+0000 7fba50359640 10 monclient: monmap: 2024-03-25T18:29:01.246 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:29:01.246 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:01.246 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:29:01.246 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:29:01.246 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:01.246 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:29:01.246 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:29:01.247 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:29:01.247 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:29:01.247 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:01.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.241+0000 7fba50359640 1 librados: starting msgr at 2024-03-25T18:29:01.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.241+0000 7fba50359640 1 librados: starting objecter 2024-03-25T18:29:01.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.241+0000 7fba50359640 1 Processor -- start 2024-03-25T18:29:01.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.241+0000 7fba50359640 1 -- start start 2024-03-25T18:29:01.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.241+0000 7fba50359640 1 librados: setting wanted keys 2024-03-25T18:29:01.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.241+0000 7fba50359640 1 librados: calling monclient init 2024-03-25T18:29:01.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.241+0000 7fba50359640 10 monclient: init 2024-03-25T18:29:01.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.242+0000 7fba50359640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:01.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.242+0000 7fba50359640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:29:01.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.242+0000 7fba50359640 10 monclient(hunting): picked mon.noname-a con 0x7fba480615c0 addr v1:172.21.15.3:6789/0 2024-03-25T18:29:01.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.242+0000 7fba50359640 10 monclient(hunting): picked mon.noname-c con 0x7fba48143b80 addr v1:172.21.15.3:6790/0 2024-03-25T18:29:01.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.242+0000 7fba50359640 10 monclient(hunting): picked mon.noname-b con 0x7fba481d5f00 addr v1:172.21.15.16:6789/0 2024-03-25T18:29:01.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.242+0000 7fba50359640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba481d5310 con 0x7fba480615c0 2024-03-25T18:29:01.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.242+0000 7fba50359640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba480acfb0 con 0x7fba481d5f00 2024-03-25T18:29:01.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.242+0000 7fba50359640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba4811cbb0 con 0x7fba48143b80 2024-03-25T18:29:01.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.242+0000 7fba50359640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:01.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.242+0000 7fba50359640 10 monclient(hunting): authenticate will time out at 7595.502573s 2024-03-25T18:29:01.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.245+0000 7fba4e0f5640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fba480615c0 0x7fba480ac8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:49640/0 (socket says 172.21.15.3:49640) 2024-03-25T18:29:01.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.245+0000 7fba4e0f5640 1 -- 172.21.15.3:0/934892433 learned_addr learned my addr 172.21.15.3:0/934892433 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:29:01.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.246+0000 7fba3effd640 10 client.?.objecter ms_handle_connect 0x7fba480615c0 2024-03-25T18:29:01.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.246+0000 7fba3effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:01.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.253+0000 7fba3effd640 10 client.?.objecter ms_handle_connect 0x7fba481d5f00 2024-03-25T18:29:01.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.253+0000 7fba3effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:01.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.253+0000 7fba3effd640 1 -- 172.21.15.3:0/934892433 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3308622131 0 0) 0x7fba481d5310 con 0x7fba480615c0 2024-03-25T18:29:01.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.253+0000 7fba3effd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:01.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.253+0000 7fba3effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:01.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.253+0000 7fba3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:01.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.253+0000 7fba3effd640 10 monclient(hunting): my global_id is 4416 2024-03-25T18:29:01.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.253+0000 7fba3effd640 1 -- 172.21.15.3:0/934892433 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba2c004eb0 con 0x7fba480615c0 2024-03-25T18:29:01.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.254+0000 7fba3effd640 10 client.?.objecter ms_handle_connect 0x7fba48143b80 2024-03-25T18:29:01.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.254+0000 7fba3effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:01.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.261+0000 7fba3effd640 1 -- 172.21.15.3:0/934892433 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1433206943 0 0) 0x7fba2c004eb0 con 0x7fba480615c0 2024-03-25T18:29:01.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.262+0000 7fba3effd640 1 -- 172.21.15.3:0/934892433 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba481d5310 con 0x7fba480615c0 2024-03-25T18:29:01.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.262+0000 7fba3effd640 1 -- 172.21.15.3:0/934892433 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fba44001a30 con 0x7fba480615c0 2024-03-25T18:29:01.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.262+0000 7fba3effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:01.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.262+0000 7fba3effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:01.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.262+0000 7fba3effd640 10 monclient(hunting): dump: 2024-03-25T18:29:01.263 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:01.263 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:01.263 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:01.264 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:01.264 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:01.264 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:01.264 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:01.264 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:01.264 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:01.264 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:01.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.262+0000 7fba3effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:01.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.263+0000 7fba3effd640 1 -- 172.21.15.3:0/934892433 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2747781931 0 0) 0x7fba4811cbb0 con 0x7fba48143b80 2024-03-25T18:29:01.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.263+0000 7fba3effd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:01.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.263+0000 7fba3effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:01.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.263+0000 7fba3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:01.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.263+0000 7fba3effd640 10 monclient(hunting): my global_id is 4382 2024-03-25T18:29:01.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.263+0000 7fba3effd640 1 -- 172.21.15.3:0/934892433 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba2c004eb0 con 0x7fba48143b80 2024-03-25T18:29:01.275 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.274+0000 7fba3effd640 1 -- 172.21.15.3:0/934892433 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3465887297 0 0) 0x7fba480acfb0 con 0x7fba481d5f00 2024-03-25T18:29:01.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.274+0000 7fba3effd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:01.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.274+0000 7fba3effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:01.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.274+0000 7fba3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:01.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.274+0000 7fba3effd640 10 monclient(hunting): my global_id is 4375 2024-03-25T18:29:01.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.274+0000 7fba3effd640 1 -- 172.21.15.3:0/934892433 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba4811cbb0 con 0x7fba481d5f00 2024-03-25T18:29:01.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.276+0000 7fba3effd640 1 -- 172.21.15.3:0/934892433 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2107034438 0 0) 0x7fba2c004eb0 con 0x7fba48143b80 2024-03-25T18:29:01.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.276+0000 7fba3effd640 1 -- 172.21.15.3:0/934892433 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba480acfb0 con 0x7fba48143b80 2024-03-25T18:29:01.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.276+0000 7fba3effd640 1 -- 172.21.15.3:0/934892433 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fba38001960 con 0x7fba48143b80 2024-03-25T18:29:01.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.276+0000 7fba3effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:01.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.276+0000 7fba3effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:01.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.276+0000 7fba3effd640 10 monclient(hunting): dump: 2024-03-25T18:29:01.278 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:01.278 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:01.278 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:01.278 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:01.278 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:01.278 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:01.279 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:01.279 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:01.279 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:01.279 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:01.284 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.282+0000 7fba3effd640 1 -- 172.21.15.3:0/934892433 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2497551833 0 0) 0x7fba481d5310 con 0x7fba480615c0 2024-03-25T18:29:01.284 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.282+0000 7fba3effd640 1 -- 172.21.15.3:0/934892433 >> v1:172.21.15.3:6790/0 conn(0x7fba48143b80 legacy=0x7fba481d57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:01.284 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.282+0000 7fba3effd640 1 -- 172.21.15.3:0/934892433 >> v1:172.21.15.16:6789/0 conn(0x7fba481d5f00 legacy=0x7fba481398a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:01.284 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.282+0000 7fba3effd640 10 monclient: _finish_hunting 0 2024-03-25T18:29:01.284 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.282+0000 7fba3effd640 1 monclient: found mon.a 2024-03-25T18:29:01.284 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.282+0000 7fba3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:01.284 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.282+0000 7fba3effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:29:01.284 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.282+0000 7fba3effd640 1 -- 172.21.15.3:0/934892433 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba480a9710 con 0x7fba480615c0 2024-03-25T18:29:01.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.282+0000 7fba3effd640 10 monclient: _finish_auth 0 2024-03-25T18:29:01.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.282+0000 7fba3effd640 10 monclient: _check_auth_tickets 2024-03-25T18:29:01.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.282+0000 7fba3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:01.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.282+0000 7fba50359640 5 monclient: authenticate success, global_id 4416 2024-03-25T18:29:01.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.282+0000 7fba50359640 10 monclient: _renew_subs 2024-03-25T18:29:01.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.282+0000 7fba50359640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:29:01.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.282+0000 7fba50359640 1 -- 172.21.15.3:0/934892433 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba480abd30 con 0x7fba480615c0 2024-03-25T18:29:01.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.282+0000 7fba50359640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:01.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.282+0000 7fba50359640 10 monclient: _renew_subs 2024-03-25T18:29:01.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.282+0000 7fba50359640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:29:01.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.282+0000 7fba50359640 1 -- 172.21.15.3:0/934892433 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba4813a190 con 0x7fba480615c0 2024-03-25T18:29:01.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.282+0000 7fba50359640 1 librados: init done 2024-03-25T18:29:01.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.284+0000 7fba50359640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:01.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.284+0000 7fba50359640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:01.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.284+0000 7fba50359640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:29:01.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.284+0000 7fba50359640 1 -- 172.21.15.3:0/934892433 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba4811ca20 con 0x7fba480615c0 2024-03-25T18:29:01.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.616+0000 7fba3effd640 1 -- 172.21.15.3:0/934892433 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba44001cb0 con 0x7fba480615c0 2024-03-25T18:29:01.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.616+0000 7fba3effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:01.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.616+0000 7fba3effd640 1 -- 172.21.15.3:0/934892433 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fba44003a60 con 0x7fba480615c0 2024-03-25T18:29:01.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.616+0000 7fba3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:01.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.616+0000 7fba3effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:01.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.616+0000 7fba3effd640 10 monclient: dump: 2024-03-25T18:29:01.619 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:01.619 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:01.619 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:01.619 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:01.619 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:01.619 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:01.619 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:01.619 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:01.619 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:01.619 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:01.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.616+0000 7fba4c8f2640 4 set_mon_vals no callback set 2024-03-25T18:29:01.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.617+0000 7fba3effd640 1 -- 172.21.15.3:0/934892433 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7fba44003c80 con 0x7fba480615c0 2024-03-25T18:29:01.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.618+0000 7fba4d8f4640 10 monclient: get_auth_request con 0x7fba2c06e020 auth_method 0 2024-03-25T18:29:01.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.618+0000 7fba3effd640 10 client.4416.objecter ms_handle_connect 0x7fba2c06e020 2024-03-25T18:29:01.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.623+0000 7fba3effd640 1 -- 172.21.15.3:0/934892433 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7fba440548e0 con 0x7fba480615c0 2024-03-25T18:29:01.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.623+0000 7fba3effd640 10 client.4416.objecter ms_dispatch 0x7fba48000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:01.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.623+0000 7fba3effd640 3 client.4416.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:01.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.623+0000 7fba3effd640 3 client.4416.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:01.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.623+0000 7fba3effd640 20 client.4416.objecter dump_active .. 0 homeless 2024-03-25T18:29:01.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.623+0000 7fba3effd640 1 -- 172.21.15.3:0/934892433 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fba440503f0 con 0x7fba480615c0 2024-03-25T18:29:01.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.623+0000 7fba3effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:01.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.623+0000 7fba3effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:01.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.845+0000 7fba50359640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:01.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.845+0000 7fba50359640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:01.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.845+0000 7fba50359640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:29:01.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.845+0000 7fba50359640 1 -- 172.21.15.3:0/934892433 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba480ac0d0 con 0x7fba480615c0 2024-03-25T18:29:01.941 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.939+0000 7fba3effd640 1 -- 172.21.15.3:0/934892433 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7fba44052320 con 0x7fba480615c0 2024-03-25T18:29:01.941 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.940+0000 7fba3effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:01.941 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.940+0000 7fba3effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:01.941 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:29:01.941 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:01.944 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.942+0000 7fba50359640 10 librados: watch_flush enter 2024-03-25T18:29:01.944 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.942+0000 7fba50359640 10 librados: watch_flush exit 2024-03-25T18:29:01.944 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.942+0000 7fba50359640 20 client.4416.objecter shutdown clearing up homeless session... 2024-03-25T18:29:01.944 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.942+0000 7fba50359640 10 client.4416.objecter successfully canceled tick 2024-03-25T18:29:01.944 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.942+0000 7fba50359640 1 -- 172.21.15.3:0/934892433 >> v1:172.21.15.3:6800/4195140966 conn(0x7fba2c06e020 legacy=0x7fba2c0704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:01.944 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.942+0000 7fba50359640 10 monclient: shutdown 2024-03-25T18:29:01.944 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.942+0000 7fba50359640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:01.944 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.942+0000 7fba50359640 1 -- 172.21.15.3:0/934892433 >> v1:172.21.15.3:6789/0 conn(0x7fba480615c0 legacy=0x7fba480ac8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:01.944 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.942+0000 7fba50359640 1 -- 172.21.15.3:0/934892433 shutdown_connections 2024-03-25T18:29:01.944 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.942+0000 7fba50359640 1 -- 172.21.15.3:0/934892433 >> 172.21.15.3:0/934892433 conn(0x7fba480b1270 msgr2=0x7fba480b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:01.944 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.942+0000 7fba50359640 1 -- 172.21.15.3:0/934892433 shutdown_connections 2024-03-25T18:29:01.944 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.942+0000 7fba50359640 1 -- 172.21.15.3:0/934892433 wait complete. 2024-03-25T18:29:01.945 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:01.942+0000 7fba50359640 1 librados: shutdown 2024-03-25T18:29:01.954 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:08.761 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:08.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.876+0000 7f19d88f4640 10 monclient: get_monmap_and_config 2024-03-25T18:29:08.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.876+0000 7f19d88f4640 10 monclient: build_initial_monmap 2024-03-25T18:29:08.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.876+0000 7f19d88f4640 10 monclient: monmap: 2024-03-25T18:29:08.877 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:29:08.877 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:08.878 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:29:08.877403+0000 2024-03-25T18:29:08.878 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:29:08.877403+0000 2024-03-25T18:29:08.878 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:08.878 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:08.878 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:29:08.878 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:29:08.878 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:29:08.878 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:08.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.876+0000 7f19d88f4640 1 Processor -- start 2024-03-25T18:29:08.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.877+0000 7f19d88f4640 1 -- start start 2024-03-25T18:29:08.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.877+0000 7f19d88f4640 10 monclient: init 2024-03-25T18:29:08.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.877+0000 7f19d88f4640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:08.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.877+0000 7f19d88f4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:29:08.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.877+0000 7f19d88f4640 10 monclient(hunting): picked mon.noname-a con 0x7f19d406f460 addr v1:172.21.15.3:6789/0 2024-03-25T18:29:08.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.877+0000 7f19d88f4640 10 monclient(hunting): picked mon.noname-b con 0x7f19d406f840 addr v1:172.21.15.3:6790/0 2024-03-25T18:29:08.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.877+0000 7f19d88f4640 10 monclient(hunting): picked mon.noname-c con 0x7f19d40728f0 addr v1:172.21.15.16:6789/0 2024-03-25T18:29:08.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.877+0000 7f19d88f4640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19d4071a40 con 0x7f19d406f460 2024-03-25T18:29:08.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.877+0000 7f19d88f4640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19d4071db0 con 0x7f19d40728f0 2024-03-25T18:29:08.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.877+0000 7f19d88f4640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19d4070650 con 0x7f19d406f840 2024-03-25T18:29:08.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.877+0000 7f19d88f4640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:08.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.877+0000 7f19d88f4640 10 monclient(hunting): authenticate will time out at 7603.138259s 2024-03-25T18:29:08.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.892+0000 7f19d1d9b640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f19d406f840 0x7f19d40721e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:56756/0 (socket says 172.21.15.3:56756) 2024-03-25T18:29:08.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.892+0000 7f19d1d9b640 1 -- 172.21.15.3:0/1244195028 learned_addr learned my addr 172.21.15.3:0/1244195028 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:29:08.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.892+0000 7f19d2d9d640 1 --1- 172.21.15.3:0/1244195028 >> v1:172.21.15.16:6789/0 conn(0x7f19d40728f0 0x7f19d414a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:37712/0 (socket says 172.21.15.3:37712) 2024-03-25T18:29:08.905 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.904+0000 7f19d159a640 1 -- 172.21.15.3:0/1244195028 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3108739270 0 0) 0x7f19d4070650 con 0x7f19d406f840 2024-03-25T18:29:08.905 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.904+0000 7f19d159a640 10 monclient(hunting): _negotiate 2024-03-25T18:29:08.905 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.904+0000 7f19d159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:08.905 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.904+0000 7f19d159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:08.905 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.904+0000 7f19d159a640 10 monclient(hunting): my global_id is 4385 2024-03-25T18:29:08.905 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.904+0000 7f19d159a640 1 -- 172.21.15.3:0/1244195028 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19bc005730 con 0x7f19d406f840 2024-03-25T18:29:08.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.908+0000 7f19d159a640 1 -- 172.21.15.3:0/1244195028 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171094844 0 0) 0x7f19d4071db0 con 0x7f19d40728f0 2024-03-25T18:29:08.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.908+0000 7f19d159a640 10 monclient(hunting): _negotiate 2024-03-25T18:29:08.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.908+0000 7f19d159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:08.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.908+0000 7f19d159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:08.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.908+0000 7f19d159a640 10 monclient(hunting): my global_id is 4378 2024-03-25T18:29:08.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.908+0000 7f19d159a640 1 -- 172.21.15.3:0/1244195028 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d4070650 con 0x7f19d40728f0 2024-03-25T18:29:08.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.913+0000 7f19d159a640 1 -- 172.21.15.3:0/1244195028 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2516663212 0 0) 0x7f19d4071a40 con 0x7f19d406f460 2024-03-25T18:29:08.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.913+0000 7f19d159a640 10 monclient(hunting): _negotiate 2024-03-25T18:29:08.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.913+0000 7f19d159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:08.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.913+0000 7f19d159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:08.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.913+0000 7f19d159a640 10 monclient(hunting): my global_id is 4419 2024-03-25T18:29:08.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.913+0000 7f19d159a640 1 -- 172.21.15.3:0/1244195028 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d4071db0 con 0x7f19d406f460 2024-03-25T18:29:08.926 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.922+0000 7f19d159a640 1 -- 172.21.15.3:0/1244195028 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4158782914 0 0) 0x7f19d4070650 con 0x7f19d40728f0 2024-03-25T18:29:08.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.922+0000 7f19d159a640 1 -- 172.21.15.3:0/1244195028 >> v1:172.21.15.3:6790/0 conn(0x7f19d406f840 legacy=0x7f19d40721e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:08.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.922+0000 7f19d159a640 1 -- 172.21.15.3:0/1244195028 >> v1:172.21.15.3:6789/0 conn(0x7f19d406f460 legacy=0x7f19d4142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:08.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.922+0000 7f19d159a640 10 monclient: _finish_hunting 0 2024-03-25T18:29:08.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.922+0000 7f19d159a640 1 monclient: found mon.noname-c 2024-03-25T18:29:08.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.922+0000 7f19d159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:08.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.922+0000 7f19d159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:29:08.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.922+0000 7f19d159a640 1 -- 172.21.15.3:0/1244195028 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d4070200 con 0x7f19d40728f0 2024-03-25T18:29:08.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.922+0000 7f19d159a640 10 monclient: _finish_auth 0 2024-03-25T18:29:08.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.922+0000 7f19d159a640 10 monclient: _check_auth_tickets 2024-03-25T18:29:08.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.922+0000 7f19d159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:08.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.922+0000 7f19d159a640 1 -- 172.21.15.3:0/1244195028 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f19cc0019f0 con 0x7f19d40728f0 2024-03-25T18:29:08.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.922+0000 7f19d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:08.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.922+0000 7f19d159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:08.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.922+0000 7f19d159a640 10 monclient: dump: 2024-03-25T18:29:08.928 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:08.928 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:08.928 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:08.928 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:08.928 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:08.928 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:08.928 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:08.928 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:08.928 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:08.928 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:08.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.922+0000 7f19d88f4640 5 monclient: authenticate success, global_id 4378 2024-03-25T18:29:08.929 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.922+0000 7f19d88f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:08.947 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.945+0000 7f19d159a640 1 -- 172.21.15.3:0/1244195028 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19cc001b50 con 0x7f19d40728f0 2024-03-25T18:29:08.947 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.945+0000 7f19d159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:08.947 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.945+0000 7f19d159a640 1 -- 172.21.15.3:0/1244195028 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f19cc001ee0 con 0x7f19d40728f0 2024-03-25T18:29:08.947 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.945+0000 7f19d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:08.947 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.946+0000 7f19d159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:08.947 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.946+0000 7f19d159a640 10 monclient: dump: 2024-03-25T18:29:08.947 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:08.947 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:08.948 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:08.948 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:08.948 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:08.948 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:08.948 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:08.948 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:08.948 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:08.948 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:08.948 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.946+0000 7f19d88f4640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:08.948 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.946+0000 7f19d88f4640 4 set_mon_vals no callback set 2024-03-25T18:29:08.950 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.946+0000 7f19d88f4640 10 monclient: shutdown 2024-03-25T18:29:08.950 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.946+0000 7f19d88f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:08.950 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.946+0000 7f19d88f4640 1 -- 172.21.15.3:0/1244195028 >> v1:172.21.15.16:6789/0 conn(0x7f19d40728f0 legacy=0x7f19d414a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:08.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.947+0000 7f19d88f4640 1 -- 172.21.15.3:0/1244195028 shutdown_connections 2024-03-25T18:29:08.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.947+0000 7f19d88f4640 1 -- 172.21.15.3:0/1244195028 >> 172.21.15.3:0/1244195028 conn(0x7f19d40b4bd0 msgr2=0x7f19d40b4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:08.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.947+0000 7f19d88f4640 1 -- 172.21.15.3:0/1244195028 shutdown_connections 2024-03-25T18:29:08.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.947+0000 7f19d88f4640 1 -- 172.21.15.3:0/1244195028 wait complete. 2024-03-25T18:29:08.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.948+0000 7f19d88f4640 10 monclient: build_initial_monmap 2024-03-25T18:29:08.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.948+0000 7f19d88f4640 10 monclient: monmap: 2024-03-25T18:29:08.951 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:29:08.951 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:08.951 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:29:08.951 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:29:08.951 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:08.951 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:29:08.951 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:29:08.951 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:29:08.952 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:29:08.952 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:08.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.948+0000 7f19d88f4640 1 librados: starting msgr at 2024-03-25T18:29:08.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.948+0000 7f19d88f4640 1 librados: starting objecter 2024-03-25T18:29:08.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.948+0000 7f19d88f4640 1 Processor -- start 2024-03-25T18:29:08.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.948+0000 7f19d88f4640 1 -- start start 2024-03-25T18:29:08.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.948+0000 7f19d88f4640 1 librados: setting wanted keys 2024-03-25T18:29:08.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.948+0000 7f19d88f4640 1 librados: calling monclient init 2024-03-25T18:29:08.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.948+0000 7f19d88f4640 10 monclient: init 2024-03-25T18:29:08.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.949+0000 7f19d88f4640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:08.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.949+0000 7f19d88f4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:29:08.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.949+0000 7f19d88f4640 10 monclient(hunting): picked mon.noname-a con 0x7f19d406f460 addr v1:172.21.15.3:6789/0 2024-03-25T18:29:08.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.949+0000 7f19d88f4640 10 monclient(hunting): picked mon.noname-c con 0x7f19d406f840 addr v1:172.21.15.3:6790/0 2024-03-25T18:29:08.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.949+0000 7f19d88f4640 10 monclient(hunting): picked mon.noname-b con 0x7f19d40728f0 addr v1:172.21.15.16:6789/0 2024-03-25T18:29:08.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.949+0000 7f19d88f4640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19d41d80b0 con 0x7f19d406f460 2024-03-25T18:29:08.953 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.949+0000 7f19d88f4640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19d41db4c0 con 0x7f19d40728f0 2024-03-25T18:29:08.953 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.949+0000 7f19d88f4640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19d4135870 con 0x7f19d406f840 2024-03-25T18:29:08.953 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.949+0000 7f19d88f4640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:08.953 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.949+0000 7f19d88f4640 10 monclient(hunting): authenticate will time out at 7603.209879s 2024-03-25T18:29:08.953 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.952+0000 7f19d2d9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f19d40728f0 0x7f19d41e31e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:37776/0 (socket says 172.21.15.3:37776) 2024-03-25T18:29:08.953 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.952+0000 7f19d2d9d640 1 -- 172.21.15.3:0/916812046 learned_addr learned my addr 172.21.15.3:0/916812046 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:29:08.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.953+0000 7f19c2ffd640 10 client.?.objecter ms_handle_connect 0x7f19d40728f0 2024-03-25T18:29:08.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.953+0000 7f19c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:08.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.957+0000 7f19c2ffd640 10 client.?.objecter ms_handle_connect 0x7f19d406f840 2024-03-25T18:29:08.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.957+0000 7f19c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:08.966 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.965+0000 7f19c2ffd640 1 -- 172.21.15.3:0/916812046 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1644545170 0 0) 0x7f19d41db4c0 con 0x7f19d40728f0 2024-03-25T18:29:08.966 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.965+0000 7f19c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:08.967 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.965+0000 7f19c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:08.967 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.965+0000 7f19c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:08.967 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.965+0000 7f19c2ffd640 10 monclient(hunting): my global_id is 4381 2024-03-25T18:29:08.967 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.965+0000 7f19c2ffd640 1 -- 172.21.15.3:0/916812046 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19b4004eb0 con 0x7f19d40728f0 2024-03-25T18:29:08.967 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.965+0000 7f19c2ffd640 1 -- 172.21.15.3:0/916812046 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 529624159 0 0) 0x7f19d4135870 con 0x7f19d406f840 2024-03-25T18:29:08.967 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.965+0000 7f19c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:08.967 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.965+0000 7f19c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:08.967 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.965+0000 7f19c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:08.967 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.965+0000 7f19c2ffd640 10 monclient(hunting): my global_id is 4388 2024-03-25T18:29:08.967 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.965+0000 7f19c2ffd640 1 -- 172.21.15.3:0/916812046 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d41db4c0 con 0x7f19d406f840 2024-03-25T18:29:08.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.969+0000 7f19c2ffd640 10 client.?.objecter ms_handle_connect 0x7f19d406f460 2024-03-25T18:29:08.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.969+0000 7f19c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:08.980 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.979+0000 7f19c2ffd640 1 -- 172.21.15.3:0/916812046 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1056637973 0 0) 0x7f19d41d80b0 con 0x7f19d406f460 2024-03-25T18:29:08.980 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.979+0000 7f19c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:08.980 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.979+0000 7f19c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:08.980 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.979+0000 7f19c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:08.980 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.979+0000 7f19c2ffd640 10 monclient(hunting): my global_id is 4422 2024-03-25T18:29:08.980 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.979+0000 7f19c2ffd640 1 -- 172.21.15.3:0/916812046 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d4135870 con 0x7f19d406f460 2024-03-25T18:29:08.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.993+0000 7f19c2ffd640 1 -- 172.21.15.3:0/916812046 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 717623515 0 0) 0x7f19d4135870 con 0x7f19d406f460 2024-03-25T18:29:08.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.993+0000 7f19c2ffd640 1 -- 172.21.15.3:0/916812046 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19d41d80b0 con 0x7f19d406f460 2024-03-25T18:29:08.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.993+0000 7f19c2ffd640 1 -- 172.21.15.3:0/916812046 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f19c40018b0 con 0x7f19d406f460 2024-03-25T18:29:08.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.993+0000 7f19c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:08.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.993+0000 7f19c2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:08.995 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.993+0000 7f19c2ffd640 10 monclient(hunting): dump: 2024-03-25T18:29:08.995 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:08.995 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:08.995 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:08.995 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:08.995 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:08.995 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:08.995 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:08.995 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:08.995 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:08.995 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:08.995 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.993+0000 7f19c2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:08.998 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.997+0000 7f19c2ffd640 1 -- 172.21.15.3:0/916812046 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2865909293 0 0) 0x7f19d41db4c0 con 0x7f19d406f840 2024-03-25T18:29:08.999 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.997+0000 7f19c2ffd640 1 -- 172.21.15.3:0/916812046 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19d4135870 con 0x7f19d406f840 2024-03-25T18:29:08.999 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.997+0000 7f19c2ffd640 1 -- 172.21.15.3:0/916812046 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f19c8001950 con 0x7f19d406f840 2024-03-25T18:29:08.999 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.997+0000 7f19c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:08.999 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.997+0000 7f19c2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:08.999 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.997+0000 7f19c2ffd640 10 monclient(hunting): dump: 2024-03-25T18:29:08.999 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:08.999 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:08.999 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:08.999 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:08.999 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:08.999 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:08.999 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:08.999 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:08.999 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:09.000 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:09.000 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.998+0000 7f19c2ffd640 1 -- 172.21.15.3:0/916812046 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1091392271 0 0) 0x7f19b4004eb0 con 0x7f19d40728f0 2024-03-25T18:29:09.000 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.998+0000 7f19c2ffd640 1 -- 172.21.15.3:0/916812046 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19d41db4c0 con 0x7f19d40728f0 2024-03-25T18:29:09.000 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.998+0000 7f19c2ffd640 1 -- 172.21.15.3:0/916812046 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f19cc001f20 con 0x7f19d40728f0 2024-03-25T18:29:09.000 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.998+0000 7f19c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:09.000 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.998+0000 7f19c2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:09.001 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:08.998+0000 7f19c2ffd640 10 monclient(hunting): dump: 2024-03-25T18:29:09.001 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:09.001 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:09.001 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:09.001 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:09.001 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:09.001 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:09.001 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:09.001 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:09.001 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:09.001 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:09.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.005+0000 7f19c2ffd640 1 -- 172.21.15.3:0/916812046 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2590536771 0 0) 0x7f19d41d80b0 con 0x7f19d406f460 2024-03-25T18:29:09.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.006+0000 7f19c2ffd640 1 -- 172.21.15.3:0/916812046 >> v1:172.21.15.3:6790/0 conn(0x7f19d406f840 legacy=0x7f19d4135160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:09.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.006+0000 7f19c2ffd640 1 -- 172.21.15.3:0/916812046 >> v1:172.21.15.16:6789/0 conn(0x7f19d40728f0 legacy=0x7f19d41e31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:09.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.006+0000 7f19c2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:29:09.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.006+0000 7f19c2ffd640 1 monclient: found mon.a 2024-03-25T18:29:09.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.006+0000 7f19c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:09.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.006+0000 7f19c2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:29:09.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.006+0000 7f19c2ffd640 1 -- 172.21.15.3:0/916812046 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d4135a00 con 0x7f19d406f460 2024-03-25T18:29:09.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.006+0000 7f19c2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:29:09.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.006+0000 7f19c2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:29:09.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.006+0000 7f19c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:09.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.006+0000 7f19d88f4640 5 monclient: authenticate success, global_id 4422 2024-03-25T18:29:09.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.006+0000 7f19d88f4640 10 monclient: _renew_subs 2024-03-25T18:29:09.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.006+0000 7f19d88f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:29:09.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.006+0000 7f19d88f4640 1 -- 172.21.15.3:0/916812046 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19d4135d70 con 0x7f19d406f460 2024-03-25T18:29:09.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.006+0000 7f19d88f4640 10 client.4422.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:09.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.006+0000 7f19d88f4640 10 monclient: _renew_subs 2024-03-25T18:29:09.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.006+0000 7f19d88f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:29:09.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.006+0000 7f19d88f4640 1 -- 172.21.15.3:0/916812046 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19d40adad0 con 0x7f19d406f460 2024-03-25T18:29:09.009 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.006+0000 7f19d88f4640 1 librados: init done 2024-03-25T18:29:09.009 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.007+0000 7f19d88f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:09.009 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.007+0000 7f19d88f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:09.009 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.007+0000 7f19d88f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:29:09.010 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.007+0000 7f19d88f4640 1 -- 172.21.15.3:0/916812046 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19d41db650 con 0x7f19d406f460 2024-03-25T18:29:09.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.314+0000 7f19c2ffd640 1 -- 172.21.15.3:0/916812046 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19c4001d20 con 0x7f19d406f460 2024-03-25T18:29:09.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.314+0000 7f19c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:09.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.314+0000 7f19c2ffd640 1 -- 172.21.15.3:0/916812046 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f19c40037b0 con 0x7f19d406f460 2024-03-25T18:29:09.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.314+0000 7f19c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:09.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.314+0000 7f19c2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:09.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.314+0000 7f19c2ffd640 10 monclient: dump: 2024-03-25T18:29:09.316 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:09.316 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:09.316 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:09.316 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:09.316 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:09.316 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:09.316 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:09.316 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:09.316 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:09.316 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:09.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.314+0000 7f19d0d99640 4 set_mon_vals no callback set 2024-03-25T18:29:09.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.315+0000 7f19c2ffd640 1 -- 172.21.15.3:0/916812046 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f19c4019800 con 0x7f19d406f460 2024-03-25T18:29:09.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.316+0000 7f19d1d9b640 10 monclient: get_auth_request con 0x7f19b406e0a0 auth_method 0 2024-03-25T18:29:09.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.316+0000 7f19c2ffd640 10 client.4422.objecter ms_handle_connect 0x7f19b406e0a0 2024-03-25T18:29:09.324 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.323+0000 7f19c2ffd640 1 -- 172.21.15.3:0/916812046 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f19c4053540 con 0x7f19d406f460 2024-03-25T18:29:09.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.323+0000 7f19c2ffd640 10 client.4422.objecter ms_dispatch 0x7f19d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:09.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.323+0000 7f19c2ffd640 3 client.4422.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:09.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.323+0000 7f19c2ffd640 3 client.4422.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:09.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.323+0000 7f19c2ffd640 20 client.4422.objecter dump_active .. 0 homeless 2024-03-25T18:29:09.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.323+0000 7f19c2ffd640 1 -- 172.21.15.3:0/916812046 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f19c404ee80 con 0x7f19d406f460 2024-03-25T18:29:09.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.323+0000 7f19c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:09.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.323+0000 7f19c2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:09.581 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.579+0000 7f19d88f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:09.581 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.580+0000 7f19d88f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:09.581 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.580+0000 7f19d88f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:29:09.581 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.580+0000 7f19d88f4640 1 -- 172.21.15.3:0/916812046 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19d4070960 con 0x7f19d406f460 2024-03-25T18:29:09.602 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.601+0000 7f19c2ffd640 1 -- 172.21.15.3:0/916812046 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f19c4056ba0 con 0x7f19d406f460 2024-03-25T18:29:09.602 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.601+0000 7f19c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:09.602 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.601+0000 7f19c2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:09.602 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:29:09.603 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:09.605 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.603+0000 7f19d88f4640 10 librados: watch_flush enter 2024-03-25T18:29:09.605 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.603+0000 7f19d88f4640 10 librados: watch_flush exit 2024-03-25T18:29:09.605 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.603+0000 7f19d88f4640 20 client.4422.objecter shutdown clearing up homeless session... 2024-03-25T18:29:09.605 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.603+0000 7f19d88f4640 10 client.4422.objecter successfully canceled tick 2024-03-25T18:29:09.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.604+0000 7f19d88f4640 1 -- 172.21.15.3:0/916812046 >> v1:172.21.15.3:6800/4195140966 conn(0x7f19b406e0a0 legacy=0x7f19b4070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:09.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.604+0000 7f19d88f4640 10 monclient: shutdown 2024-03-25T18:29:09.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.604+0000 7f19d88f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:09.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.604+0000 7f19d88f4640 1 -- 172.21.15.3:0/916812046 >> v1:172.21.15.3:6789/0 conn(0x7f19d406f460 legacy=0x7f19d41d79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:09.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.604+0000 7f19d88f4640 1 -- 172.21.15.3:0/916812046 shutdown_connections 2024-03-25T18:29:09.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.604+0000 7f19d88f4640 1 -- 172.21.15.3:0/916812046 >> 172.21.15.3:0/916812046 conn(0x7f19d40b4bd0 msgr2=0x7f19d40affd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:09.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.604+0000 7f19d88f4640 1 -- 172.21.15.3:0/916812046 shutdown_connections 2024-03-25T18:29:09.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.604+0000 7f19d88f4640 1 -- 172.21.15.3:0/916812046 wait complete. 2024-03-25T18:29:09.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:09.604+0000 7f19d88f4640 1 librados: shutdown 2024-03-25T18:29:09.616 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:16.423 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:16.545 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.544+0000 7f45a4b1c640 10 monclient: get_monmap_and_config 2024-03-25T18:29:16.545 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.544+0000 7f45a4b1c640 10 monclient: build_initial_monmap 2024-03-25T18:29:16.545 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.544+0000 7f45a4b1c640 10 monclient: monmap: 2024-03-25T18:29:16.546 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:29:16.546 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:16.546 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:29:16.545218+0000 2024-03-25T18:29:16.546 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:29:16.545218+0000 2024-03-25T18:29:16.546 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:16.546 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:16.546 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:29:16.546 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:29:16.546 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:29:16.546 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:16.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.545+0000 7f45a4b1c640 1 Processor -- start 2024-03-25T18:29:16.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.545+0000 7f45a4b1c640 1 -- start start 2024-03-25T18:29:16.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.545+0000 7f45a4b1c640 10 monclient: init 2024-03-25T18:29:16.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.545+0000 7f45a4b1c640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:16.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.545+0000 7f45a4b1c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:29:16.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.545+0000 7f45a4b1c640 10 monclient(hunting): picked mon.noname-c con 0x7f45a00b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:29:16.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.545+0000 7f45a4b1c640 10 monclient(hunting): picked mon.noname-b con 0x7f45a00b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:29:16.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.545+0000 7f45a4b1c640 10 monclient(hunting): picked mon.noname-a con 0x7f45a013c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:29:16.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.545+0000 7f45a4b1c640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a00ab0e0 con 0x7f45a013c480 2024-03-25T18:29:16.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.545+0000 7f45a4b1c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a00ab3c0 con 0x7f45a00b4ff0 2024-03-25T18:29:16.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.545+0000 7f45a4b1c640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a00a9f30 con 0x7f45a00b4670 2024-03-25T18:29:16.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.545+0000 7f45a4b1c640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:16.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.545+0000 7f45a4b1c640 10 monclient(hunting): authenticate will time out at 7610.806291s 2024-03-25T18:29:16.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.559+0000 7f459e59c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f45a00b4ff0 0x7f45a013bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:42410/0 (socket says 172.21.15.3:42410) 2024-03-25T18:29:16.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.559+0000 7f459e59c640 1 -- 172.21.15.3:0/197095337 learned_addr learned my addr 172.21.15.3:0/197095337 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:29:16.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.568+0000 7f459dd9b640 1 -- 172.21.15.3:0/197095337 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1319700853 0 0) 0x7f45a00a9f30 con 0x7f45a00b4670 2024-03-25T18:29:16.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.568+0000 7f459dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:29:16.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.568+0000 7f459dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:16.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.568+0000 7f459dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:16.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.568+0000 7f459dd9b640 10 monclient(hunting): my global_id is 4391 2024-03-25T18:29:16.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.568+0000 7f459dd9b640 1 -- 172.21.15.3:0/197095337 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f458c003520 con 0x7f45a00b4670 2024-03-25T18:29:16.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.574+0000 7f459dd9b640 1 -- 172.21.15.3:0/197095337 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3587844850 0 0) 0x7f45a00ab3c0 con 0x7f45a00b4ff0 2024-03-25T18:29:16.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.574+0000 7f459dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:29:16.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.574+0000 7f459dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:16.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.574+0000 7f459dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:16.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.574+0000 7f459dd9b640 10 monclient(hunting): my global_id is 4384 2024-03-25T18:29:16.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.574+0000 7f459dd9b640 1 -- 172.21.15.3:0/197095337 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45a00a9f30 con 0x7f45a00b4ff0 2024-03-25T18:29:16.581 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.580+0000 7f459dd9b640 1 -- 172.21.15.3:0/197095337 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4081920899 0 0) 0x7f458c003520 con 0x7f45a00b4670 2024-03-25T18:29:16.581 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.580+0000 7f459dd9b640 1 -- 172.21.15.3:0/197095337 >> v1:172.21.15.16:6789/0 conn(0x7f45a00b4ff0 legacy=0x7f45a013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:16.582 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.580+0000 7f459dd9b640 1 -- 172.21.15.3:0/197095337 >> v1:172.21.15.3:6789/0 conn(0x7f45a013c480 legacy=0x7f45a014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:16.582 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.580+0000 7f459dd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:29:16.582 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.580+0000 7f459dd9b640 1 monclient: found mon.noname-b 2024-03-25T18:29:16.582 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.580+0000 7f459dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:16.582 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.580+0000 7f459dd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-25T18:29:16.582 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.580+0000 7f459dd9b640 1 -- 172.21.15.3:0/197095337 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a00a9b70 con 0x7f45a00b4670 2024-03-25T18:29:16.582 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.580+0000 7f459dd9b640 10 monclient: _finish_auth 0 2024-03-25T18:29:16.582 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.580+0000 7f459dd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:29:16.582 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.580+0000 7f459dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:16.582 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.580+0000 7f459dd9b640 1 -- 172.21.15.3:0/197095337 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4588001690 con 0x7f45a00b4670 2024-03-25T18:29:16.582 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.580+0000 7f459dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:16.582 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.580+0000 7f459dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:16.582 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.580+0000 7f459dd9b640 10 monclient: dump: 2024-03-25T18:29:16.583 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:16.583 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:16.583 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:16.583 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:16.583 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:16.583 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:16.583 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:16.583 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:16.583 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:16.583 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:16.583 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.580+0000 7f45a4b1c640 5 monclient: authenticate success, global_id 4391 2024-03-25T18:29:16.583 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.580+0000 7f45a4b1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:16.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.605+0000 7f459dd9b640 1 -- 172.21.15.3:0/197095337 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45880017f0 con 0x7f45a00b4670 2024-03-25T18:29:16.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.605+0000 7f459dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:16.607 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.605+0000 7f459dd9b640 1 -- 172.21.15.3:0/197095337 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4588001c00 con 0x7f45a00b4670 2024-03-25T18:29:16.607 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.605+0000 7f45a4b1c640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:16.607 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.605+0000 7f45a4b1c640 4 set_mon_vals no callback set 2024-03-25T18:29:16.607 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.605+0000 7f459dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:16.607 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.605+0000 7f459dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:16.607 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.605+0000 7f459dd9b640 10 monclient: dump: 2024-03-25T18:29:16.607 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:16.607 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:16.607 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:16.607 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:16.607 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:16.607 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:16.607 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:16.607 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:16.607 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:16.608 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:16.609 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.606+0000 7f45a4b1c640 10 monclient: shutdown 2024-03-25T18:29:16.609 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.606+0000 7f45a4b1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:16.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.606+0000 7f45a4b1c640 1 -- 172.21.15.3:0/197095337 >> v1:172.21.15.3:6790/0 conn(0x7f45a00b4670 legacy=0x7f45a01425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:16.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.606+0000 7f45a4b1c640 1 -- 172.21.15.3:0/197095337 shutdown_connections 2024-03-25T18:29:16.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.606+0000 7f45a4b1c640 1 -- 172.21.15.3:0/197095337 >> 172.21.15.3:0/197095337 conn(0x7f45a00617a0 msgr2=0x7f45a0061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:16.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.606+0000 7f45a4b1c640 1 -- 172.21.15.3:0/197095337 shutdown_connections 2024-03-25T18:29:16.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.606+0000 7f45a4b1c640 1 -- 172.21.15.3:0/197095337 wait complete. 2024-03-25T18:29:16.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.607+0000 7f45a4b1c640 10 monclient: build_initial_monmap 2024-03-25T18:29:16.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.607+0000 7f45a4b1c640 10 monclient: monmap: 2024-03-25T18:29:16.610 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:29:16.610 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:16.610 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:29:16.610 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:29:16.610 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:16.610 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:29:16.611 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:29:16.611 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:29:16.611 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:29:16.611 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:16.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.607+0000 7f45a4b1c640 1 librados: starting msgr at 2024-03-25T18:29:16.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.607+0000 7f45a4b1c640 1 librados: starting objecter 2024-03-25T18:29:16.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.607+0000 7f45a4b1c640 1 Processor -- start 2024-03-25T18:29:16.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.607+0000 7f45a4b1c640 1 -- start start 2024-03-25T18:29:16.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.607+0000 7f45a4b1c640 1 librados: setting wanted keys 2024-03-25T18:29:16.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.607+0000 7f45a4b1c640 1 librados: calling monclient init 2024-03-25T18:29:16.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.607+0000 7f45a4b1c640 10 monclient: init 2024-03-25T18:29:16.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.608+0000 7f45a4b1c640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:16.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.608+0000 7f45a4b1c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:29:16.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.608+0000 7f45a4b1c640 10 monclient(hunting): picked mon.noname-b con 0x7f45a00b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:29:16.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.608+0000 7f45a4b1c640 10 monclient(hunting): picked mon.noname-a con 0x7f45a00b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:29:16.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.608+0000 7f45a4b1c640 10 monclient(hunting): picked mon.noname-c con 0x7f45a013c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:29:16.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.608+0000 7f45a4b1c640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a013b120 con 0x7f45a00b4ff0 2024-03-25T18:29:16.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.608+0000 7f45a4b1c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a00b1990 con 0x7f45a00b4670 2024-03-25T18:29:16.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.608+0000 7f45a4b1c640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a00b0130 con 0x7f45a013c480 2024-03-25T18:29:16.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.608+0000 7f45a4b1c640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:16.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.608+0000 7f45a4b1c640 10 monclient(hunting): authenticate will time out at 7610.868679s 2024-03-25T18:29:16.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.617+0000 7f459ed9d640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f45a013c480 0x7f45a00aeaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:33488/0 (socket says 172.21.15.3:33488) 2024-03-25T18:29:16.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.617+0000 7f459ed9d640 1 -- 172.21.15.3:0/3852917377 learned_addr learned my addr 172.21.15.3:0/3852917377 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:29:16.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.619+0000 7f4596ffd640 10 client.?.objecter ms_handle_connect 0x7f45a013c480 2024-03-25T18:29:16.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.619+0000 7f4596ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:16.621 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.620+0000 7f4596ffd640 10 client.?.objecter ms_handle_connect 0x7f45a00b4670 2024-03-25T18:29:16.621 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.620+0000 7f4596ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:16.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.627+0000 7f4596ffd640 1 -- 172.21.15.3:0/3852917377 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3314584940 0 0) 0x7f45a00b0130 con 0x7f45a013c480 2024-03-25T18:29:16.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.627+0000 7f4596ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:16.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.627+0000 7f4596ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:16.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.627+0000 7f4596ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:16.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.627+0000 7f4596ffd640 10 monclient(hunting): my global_id is 4394 2024-03-25T18:29:16.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.627+0000 7f4596ffd640 1 -- 172.21.15.3:0/3852917377 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4580004eb0 con 0x7f45a013c480 2024-03-25T18:29:16.641 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.640+0000 7f4596ffd640 1 -- 172.21.15.3:0/3852917377 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3393222908 0 0) 0x7f45a00b1990 con 0x7f45a00b4670 2024-03-25T18:29:16.641 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.640+0000 7f4596ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:16.642 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.640+0000 7f4596ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:16.642 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.640+0000 7f4596ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:16.642 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.640+0000 7f4596ffd640 10 monclient(hunting): my global_id is 4387 2024-03-25T18:29:16.642 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.640+0000 7f4596ffd640 1 -- 172.21.15.3:0/3852917377 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45a00b0130 con 0x7f45a00b4670 2024-03-25T18:29:16.645 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.644+0000 7f4596ffd640 1 -- 172.21.15.3:0/3852917377 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1490431075 0 0) 0x7f4580004eb0 con 0x7f45a013c480 2024-03-25T18:29:16.645 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.644+0000 7f4596ffd640 1 -- 172.21.15.3:0/3852917377 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45a00b1990 con 0x7f45a013c480 2024-03-25T18:29:16.645 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.644+0000 7f4596ffd640 1 -- 172.21.15.3:0/3852917377 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4598001c90 con 0x7f45a013c480 2024-03-25T18:29:16.645 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.644+0000 7f4596ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:16.645 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.644+0000 7f4596ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:16.645 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.644+0000 7f4596ffd640 10 monclient(hunting): dump: 2024-03-25T18:29:16.645 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:16.645 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:16.645 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:16.646 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:16.646 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:16.646 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:16.646 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:16.646 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:16.646 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:16.646 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:16.646 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.644+0000 7f4596ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:16.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.649+0000 7f4596ffd640 10 client.?.objecter ms_handle_connect 0x7f45a00b4ff0 2024-03-25T18:29:16.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.649+0000 7f4596ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:16.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.658+0000 7f4596ffd640 1 -- 172.21.15.3:0/3852917377 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2793816233 0 0) 0x7f45a00b1990 con 0x7f45a013c480 2024-03-25T18:29:16.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.658+0000 7f4596ffd640 1 -- 172.21.15.3:0/3852917377 >> v1:172.21.15.16:6789/0 conn(0x7f45a00b4670 legacy=0x7f45a00b0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:16.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.658+0000 7f4596ffd640 1 -- 172.21.15.3:0/3852917377 >> v1:172.21.15.3:6789/0 conn(0x7f45a00b4ff0 legacy=0x7f45a00ae390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:16.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.658+0000 7f4596ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:29:16.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.658+0000 7f4596ffd640 1 monclient: found mon.c 2024-03-25T18:29:16.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.658+0000 7f4596ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:16.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.658+0000 7f4596ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:29:16.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.658+0000 7f4596ffd640 1 -- 172.21.15.3:0/3852917377 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a00b0410 con 0x7f45a013c480 2024-03-25T18:29:16.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.658+0000 7f4596ffd640 10 monclient: _finish_auth 0 2024-03-25T18:29:16.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.658+0000 7f4596ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:29:16.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.658+0000 7f4596ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:16.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.658+0000 7f45a4b1c640 5 monclient: authenticate success, global_id 4394 2024-03-25T18:29:16.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.658+0000 7f45a4b1c640 10 monclient: _renew_subs 2024-03-25T18:29:16.661 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.658+0000 7f45a4b1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:29:16.661 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.658+0000 7f45a4b1c640 1 -- 172.21.15.3:0/3852917377 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45a00b06f0 con 0x7f45a013c480 2024-03-25T18:29:16.661 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.658+0000 7f45a4b1c640 10 client.4394.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:16.661 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.658+0000 7f45a4b1c640 10 monclient: _renew_subs 2024-03-25T18:29:16.661 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.658+0000 7f45a4b1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:29:16.661 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.658+0000 7f45a4b1c640 1 -- 172.21.15.3:0/3852917377 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f45a01e35a0 con 0x7f45a013c480 2024-03-25T18:29:16.661 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.658+0000 7f45a4b1c640 1 librados: init done 2024-03-25T18:29:16.661 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.659+0000 7f45a4b1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:16.662 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.659+0000 7f45a4b1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:16.662 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.659+0000 7f45a4b1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:29:16.662 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:16.659+0000 7f45a4b1c640 1 -- 172.21.15.3:0/3852917377 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45a00688c0 con 0x7f45a013c480 2024-03-25T18:29:17.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.002+0000 7f4596ffd640 1 -- 172.21.15.3:0/3852917377 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4598003990 con 0x7f45a013c480 2024-03-25T18:29:17.004 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.002+0000 7f4596ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:17.004 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.002+0000 7f4596ffd640 1 -- 172.21.15.3:0/3852917377 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4598003d60 con 0x7f45a013c480 2024-03-25T18:29:17.004 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.002+0000 7f4596ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:17.004 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.002+0000 7f4596ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:17.004 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.002+0000 7f4596ffd640 10 monclient: dump: 2024-03-25T18:29:17.004 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:17.004 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:17.004 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:17.004 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:17.004 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:17.004 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:17.004 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:17.005 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:17.005 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:17.005 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:17.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.002+0000 7f459d59a640 4 set_mon_vals no callback set 2024-03-25T18:29:17.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.003+0000 7f4596ffd640 1 -- 172.21.15.3:0/3852917377 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f4598003f80 con 0x7f45a013c480 2024-03-25T18:29:17.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.003+0000 7f459e59c640 10 monclient: get_auth_request con 0x7f458006dd10 auth_method 0 2024-03-25T18:29:17.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.004+0000 7f4596ffd640 10 client.4394.objecter ms_handle_connect 0x7f458006dd10 2024-03-25T18:29:17.010 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.009+0000 7f4596ffd640 1 -- 172.21.15.3:0/3852917377 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f4598054b40 con 0x7f45a013c480 2024-03-25T18:29:17.010 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.009+0000 7f4596ffd640 10 client.4394.objecter ms_dispatch 0x7f45a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:17.010 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.009+0000 7f4596ffd640 3 client.4394.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:17.010 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.009+0000 7f4596ffd640 3 client.4394.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:17.010 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.009+0000 7f4596ffd640 20 client.4394.objecter dump_active .. 0 homeless 2024-03-25T18:29:17.010 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.009+0000 7f4596ffd640 1 -- 172.21.15.3:0/3852917377 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4598050650 con 0x7f45a013c480 2024-03-25T18:29:17.010 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.009+0000 7f4596ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:17.011 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.009+0000 7f4596ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:17.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.251+0000 7f45a4b1c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:17.253 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.251+0000 7f45a4b1c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:17.253 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.251+0000 7f45a4b1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:29:17.253 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.251+0000 7f45a4b1c640 1 -- 172.21.15.3:0/3852917377 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45a006dbd0 con 0x7f45a013c480 2024-03-25T18:29:17.292 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.291+0000 7f4596ffd640 1 -- 172.21.15.3:0/3852917377 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f4598052580 con 0x7f45a013c480 2024-03-25T18:29:17.292 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.291+0000 7f4596ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:17.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.291+0000 7f4596ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:17.293 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:29:17.293 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:17.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.293+0000 7f45a4b1c640 10 librados: watch_flush enter 2024-03-25T18:29:17.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.293+0000 7f45a4b1c640 10 librados: watch_flush exit 2024-03-25T18:29:17.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.293+0000 7f45a4b1c640 20 client.4394.objecter shutdown clearing up homeless session... 2024-03-25T18:29:17.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.293+0000 7f45a4b1c640 10 client.4394.objecter successfully canceled tick 2024-03-25T18:29:17.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.294+0000 7f45a4b1c640 1 -- 172.21.15.3:0/3852917377 >> v1:172.21.15.3:6800/4195140966 conn(0x7f458006dd10 legacy=0x7f45800701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:17.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.294+0000 7f45a4b1c640 10 monclient: shutdown 2024-03-25T18:29:17.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.294+0000 7f45a4b1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:17.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.294+0000 7f45a4b1c640 1 -- 172.21.15.3:0/3852917377 >> v1:172.21.15.3:6790/0 conn(0x7f45a013c480 legacy=0x7f45a00aeaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:17.296 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.294+0000 7f45a4b1c640 1 -- 172.21.15.3:0/3852917377 shutdown_connections 2024-03-25T18:29:17.296 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.294+0000 7f45a4b1c640 1 -- 172.21.15.3:0/3852917377 >> 172.21.15.3:0/3852917377 conn(0x7f45a00617a0 msgr2=0x7f45a0145670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:17.296 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.294+0000 7f45a4b1c640 1 -- 172.21.15.3:0/3852917377 shutdown_connections 2024-03-25T18:29:17.296 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.294+0000 7f45a4b1c640 1 -- 172.21.15.3:0/3852917377 wait complete. 2024-03-25T18:29:17.296 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:17.294+0000 7f45a4b1c640 1 librados: shutdown 2024-03-25T18:29:17.307 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:21.755 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:04:35.484 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:21.761 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:04:35.490 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:21.801 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:04:35.530 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:21.813 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:04:35.505 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:21.821 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:04:35.513 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:21.821 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:04:35.550 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:21.834 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:04:35.564 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:21.858 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:04:35.550 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:21.860 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:04:35.590 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:24.115 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:24.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.229+0000 7f498e96b640 10 monclient: get_monmap_and_config 2024-03-25T18:29:24.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.229+0000 7f498e96b640 10 monclient: build_initial_monmap 2024-03-25T18:29:24.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.229+0000 7f498e96b640 10 monclient: monmap: 2024-03-25T18:29:24.231 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:29:24.231 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:24.231 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:29:24.231025+0000 2024-03-25T18:29:24.231 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:29:24.231025+0000 2024-03-25T18:29:24.231 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:24.231 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:24.231 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:29:24.231 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:29:24.231 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:29:24.231 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:24.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.230+0000 7f498e96b640 1 Processor -- start 2024-03-25T18:29:24.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.230+0000 7f498e96b640 1 -- start start 2024-03-25T18:29:24.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.230+0000 7f498e96b640 10 monclient: init 2024-03-25T18:29:24.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.231+0000 7f498e96b640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:24.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.231+0000 7f498e96b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:29:24.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.231+0000 7f498e96b640 10 monclient(hunting): picked mon.noname-c con 0x7f49880b4e70 addr v1:172.21.15.16:6789/0 2024-03-25T18:29:24.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.231+0000 7f498e96b640 10 monclient(hunting): picked mon.noname-b con 0x7f49880b44b0 addr v1:172.21.15.3:6790/0 2024-03-25T18:29:24.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.231+0000 7f498e96b640 10 monclient(hunting): picked mon.noname-a con 0x7f49880b1b00 addr v1:172.21.15.3:6789/0 2024-03-25T18:29:24.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.231+0000 7f498e96b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49880a96f0 con 0x7f49880b1b00 2024-03-25T18:29:24.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.231+0000 7f498e96b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49880ab0e0 con 0x7f49880b4e70 2024-03-25T18:29:24.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.231+0000 7f498e96b640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4988145f70 con 0x7f49880b44b0 2024-03-25T18:29:24.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.231+0000 7f498e96b640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:24.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.231+0000 7f498e96b640 10 monclient(hunting): authenticate will time out at 7618.491795s 2024-03-25T18:29:24.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.241+0000 7f49877fe640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f49880b44b0 0x7f4988145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:48370/0 (socket says 172.21.15.3:48370) 2024-03-25T18:29:24.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.241+0000 7f49877fe640 1 -- 172.21.15.3:0/2023318813 learned_addr learned my addr 172.21.15.3:0/2023318813 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:29:24.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.253+0000 7f4986ffd640 1 -- 172.21.15.3:0/2023318813 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1607310751 0 0) 0x7f49880ab0e0 con 0x7f49880b4e70 2024-03-25T18:29:24.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.253+0000 7f4986ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:24.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.253+0000 7f4986ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:24.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.253+0000 7f4986ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:24.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.253+0000 7f4986ffd640 10 monclient(hunting): my global_id is 4390 2024-03-25T18:29:24.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.253+0000 7f4986ffd640 1 -- 172.21.15.3:0/2023318813 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4970003520 con 0x7f49880b4e70 2024-03-25T18:29:24.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.257+0000 7f4986ffd640 1 -- 172.21.15.3:0/2023318813 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 927029946 0 0) 0x7f4988145f70 con 0x7f49880b44b0 2024-03-25T18:29:24.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.257+0000 7f4986ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:24.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.257+0000 7f4986ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:24.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.257+0000 7f4986ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:24.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.257+0000 7f4986ffd640 10 monclient(hunting): my global_id is 4397 2024-03-25T18:29:24.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.257+0000 7f4986ffd640 1 -- 172.21.15.3:0/2023318813 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49880ab0e0 con 0x7f49880b44b0 2024-03-25T18:29:24.270 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.269+0000 7f4986ffd640 1 -- 172.21.15.3:0/2023318813 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4065650552 0 0) 0x7f4970003520 con 0x7f49880b4e70 2024-03-25T18:29:24.270 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.269+0000 7f4986ffd640 1 -- 172.21.15.3:0/2023318813 >> v1:172.21.15.3:6790/0 conn(0x7f49880b44b0 legacy=0x7f4988145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:24.271 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.269+0000 7f4986ffd640 1 -- 172.21.15.3:0/2023318813 >> v1:172.21.15.3:6789/0 conn(0x7f49880b1b00 legacy=0x7f498814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:24.271 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.269+0000 7f4986ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:29:24.271 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.269+0000 7f4986ffd640 1 monclient: found mon.noname-c 2024-03-25T18:29:24.271 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.269+0000 7f4986ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:24.271 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.269+0000 7f4986ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:29:24.271 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.269+0000 7f4986ffd640 1 -- 172.21.15.3:0/2023318813 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49880a9b80 con 0x7f49880b4e70 2024-03-25T18:29:24.271 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.269+0000 7f4986ffd640 10 monclient: _finish_auth 0 2024-03-25T18:29:24.271 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.269+0000 7f4986ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:29:24.271 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.269+0000 7f4986ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:24.271 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.269+0000 7f4986ffd640 1 -- 172.21.15.3:0/2023318813 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4978001690 con 0x7f49880b4e70 2024-03-25T18:29:24.271 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.269+0000 7f4986ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:24.271 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.269+0000 7f4986ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:24.271 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.269+0000 7f4986ffd640 10 monclient: dump: 2024-03-25T18:29:24.271 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:24.271 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:24.272 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:24.272 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:24.272 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:24.272 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:24.272 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:24.272 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:24.272 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:24.272 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:24.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.269+0000 7f498e96b640 5 monclient: authenticate success, global_id 4390 2024-03-25T18:29:24.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.269+0000 7f498e96b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:24.283 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.282+0000 7f4986ffd640 1 -- 172.21.15.3:0/2023318813 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49780017f0 con 0x7f49880b4e70 2024-03-25T18:29:24.283 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.282+0000 7f4986ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:24.283 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.282+0000 7f4986ffd640 1 -- 172.21.15.3:0/2023318813 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4978001c00 con 0x7f49880b4e70 2024-03-25T18:29:24.283 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.282+0000 7f498e96b640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:24.284 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.282+0000 7f498e96b640 4 set_mon_vals no callback set 2024-03-25T18:29:24.284 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.282+0000 7f4986ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:24.284 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.282+0000 7f4986ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:24.284 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.282+0000 7f4986ffd640 10 monclient: dump: 2024-03-25T18:29:24.284 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:24.284 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:24.284 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:24.284 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:24.284 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:24.284 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:24.284 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:24.285 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:24.285 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:24.285 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:24.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.283+0000 7f498e96b640 10 monclient: shutdown 2024-03-25T18:29:24.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.283+0000 7f498e96b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:24.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.283+0000 7f498e96b640 1 -- 172.21.15.3:0/2023318813 >> v1:172.21.15.16:6789/0 conn(0x7f49880b4e70 legacy=0x7f498813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:24.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.283+0000 7f498e96b640 1 -- 172.21.15.3:0/2023318813 shutdown_connections 2024-03-25T18:29:24.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.283+0000 7f498e96b640 1 -- 172.21.15.3:0/2023318813 >> 172.21.15.3:0/2023318813 conn(0x7f49880617a0 msgr2=0x7f4988061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:24.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.283+0000 7f498e96b640 1 -- 172.21.15.3:0/2023318813 shutdown_connections 2024-03-25T18:29:24.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.283+0000 7f498e96b640 1 -- 172.21.15.3:0/2023318813 wait complete. 2024-03-25T18:29:24.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.284+0000 7f498e96b640 10 monclient: build_initial_monmap 2024-03-25T18:29:24.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.284+0000 7f498e96b640 10 monclient: monmap: 2024-03-25T18:29:24.288 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:29:24.288 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:24.288 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:29:24.288 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:29:24.288 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:24.288 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:29:24.288 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:29:24.288 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:29:24.289 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:29:24.289 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:24.289 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.284+0000 7f498e96b640 1 librados: starting msgr at 2024-03-25T18:29:24.289 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.284+0000 7f498e96b640 1 librados: starting objecter 2024-03-25T18:29:24.289 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.284+0000 7f498e96b640 1 Processor -- start 2024-03-25T18:29:24.289 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.285+0000 7f498e96b640 1 -- start start 2024-03-25T18:29:24.289 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.285+0000 7f498e96b640 1 librados: setting wanted keys 2024-03-25T18:29:24.289 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.285+0000 7f498e96b640 1 librados: calling monclient init 2024-03-25T18:29:24.289 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.285+0000 7f498e96b640 10 monclient: init 2024-03-25T18:29:24.289 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.285+0000 7f498e96b640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:24.289 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.285+0000 7f498e96b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:29:24.289 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.285+0000 7f498e96b640 10 monclient(hunting): picked mon.noname-c con 0x7f49880b1b00 addr v1:172.21.15.3:6790/0 2024-03-25T18:29:24.289 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.285+0000 7f498e96b640 10 monclient(hunting): picked mon.noname-b con 0x7f49880b44b0 addr v1:172.21.15.16:6789/0 2024-03-25T18:29:24.289 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.285+0000 7f498e96b640 10 monclient(hunting): picked mon.noname-a con 0x7f49880b4e70 addr v1:172.21.15.3:6789/0 2024-03-25T18:29:24.290 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.285+0000 7f498e96b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f498813bd00 con 0x7f49880b4e70 2024-03-25T18:29:24.290 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.285+0000 7f498e96b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4988145670 con 0x7f49880b44b0 2024-03-25T18:29:24.290 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.285+0000 7f498e96b640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f498813cb10 con 0x7f49880b1b00 2024-03-25T18:29:24.290 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.285+0000 7f498e96b640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:24.290 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.285+0000 7f498e96b640 10 monclient(hunting): authenticate will time out at 7618.545997s 2024-03-25T18:29:24.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.293+0000 7f4987fff640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f49880b1b00 0x7f4988129a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:48428/0 (socket says 172.21.15.3:48428) 2024-03-25T18:29:24.294 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.293+0000 7f4987fff640 1 -- 172.21.15.3:0/571353817 learned_addr learned my addr 172.21.15.3:0/571353817 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:29:24.298 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.297+0000 7f4984ff9640 10 client.?.objecter ms_handle_connect 0x7f49880b1b00 2024-03-25T18:29:24.298 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.297+0000 7f4984ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:24.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.303+0000 7f4984ff9640 10 client.?.objecter ms_handle_connect 0x7f49880b44b0 2024-03-25T18:29:24.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.303+0000 7f4984ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:24.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.309+0000 7f4984ff9640 1 -- 172.21.15.3:0/571353817 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1000007229 0 0) 0x7f4988145670 con 0x7f49880b44b0 2024-03-25T18:29:24.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.309+0000 7f4984ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:29:24.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.309+0000 7f4984ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:24.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.309+0000 7f4984ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:24.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.309+0000 7f4984ff9640 10 monclient(hunting): my global_id is 4393 2024-03-25T18:29:24.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.309+0000 7f4984ff9640 1 -- 172.21.15.3:0/571353817 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4974004eb0 con 0x7f49880b44b0 2024-03-25T18:29:24.311 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.311+0000 7f4984ff9640 10 client.?.objecter ms_handle_connect 0x7f49880b4e70 2024-03-25T18:29:24.312 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.311+0000 7f4984ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:24.312 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.311+0000 7f4984ff9640 1 -- 172.21.15.3:0/571353817 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1121918731 0 0) 0x7f498813cb10 con 0x7f49880b1b00 2024-03-25T18:29:24.312 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.311+0000 7f4984ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:29:24.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.311+0000 7f4984ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:24.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.311+0000 7f4984ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:24.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.311+0000 7f4984ff9640 10 monclient(hunting): my global_id is 4400 2024-03-25T18:29:24.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.311+0000 7f4984ff9640 1 -- 172.21.15.3:0/571353817 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4988145670 con 0x7f49880b1b00 2024-03-25T18:29:24.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.326+0000 7f4984ff9640 1 -- 172.21.15.3:0/571353817 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 598880526 0 0) 0x7f4974004eb0 con 0x7f49880b44b0 2024-03-25T18:29:24.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.326+0000 7f4984ff9640 1 -- 172.21.15.3:0/571353817 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f498813cb10 con 0x7f49880b44b0 2024-03-25T18:29:24.328 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.326+0000 7f4984ff9640 1 -- 172.21.15.3:0/571353817 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f497c001930 con 0x7f49880b44b0 2024-03-25T18:29:24.328 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.326+0000 7f4984ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:24.328 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.326+0000 7f4984ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:24.328 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.326+0000 7f4984ff9640 10 monclient(hunting): dump: 2024-03-25T18:29:24.328 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:24.328 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:24.328 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:24.328 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:24.328 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:24.328 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:24.328 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:24.328 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:24.328 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:24.328 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:24.328 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.326+0000 7f4984ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:24.329 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.328+0000 7f4984ff9640 1 -- 172.21.15.3:0/571353817 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3861574937 0 0) 0x7f498813bd00 con 0x7f49880b4e70 2024-03-25T18:29:24.329 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.328+0000 7f4984ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:29:24.329 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.328+0000 7f4984ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:24.329 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.328+0000 7f4984ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:24.329 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.328+0000 7f4984ff9640 10 monclient(hunting): my global_id is 4431 2024-03-25T18:29:24.329 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.328+0000 7f4984ff9640 1 -- 172.21.15.3:0/571353817 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4974004eb0 con 0x7f49880b4e70 2024-03-25T18:29:24.339 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.338+0000 7f4984ff9640 1 -- 172.21.15.3:0/571353817 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3786477572 0 0) 0x7f4974004eb0 con 0x7f49880b4e70 2024-03-25T18:29:24.339 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.338+0000 7f4984ff9640 1 -- 172.21.15.3:0/571353817 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f498813bd00 con 0x7f49880b4e70 2024-03-25T18:29:24.339 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.338+0000 7f4984ff9640 1 -- 172.21.15.3:0/571353817 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4980001b70 con 0x7f49880b4e70 2024-03-25T18:29:24.340 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.338+0000 7f4984ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:24.340 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.338+0000 7f4984ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:24.340 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.338+0000 7f4984ff9640 10 monclient(hunting): dump: 2024-03-25T18:29:24.340 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:24.340 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:24.340 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:24.340 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:24.340 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:24.340 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:24.340 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:24.340 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:24.340 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:24.340 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:24.341 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.339+0000 7f4984ff9640 1 -- 172.21.15.3:0/571353817 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 985398938 0 0) 0x7f4988145670 con 0x7f49880b1b00 2024-03-25T18:29:24.341 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.339+0000 7f4984ff9640 1 -- 172.21.15.3:0/571353817 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4974004eb0 con 0x7f49880b1b00 2024-03-25T18:29:24.341 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.339+0000 7f4984ff9640 1 -- 172.21.15.3:0/571353817 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4978001b40 con 0x7f49880b1b00 2024-03-25T18:29:24.341 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.339+0000 7f4984ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:24.341 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.339+0000 7f4984ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:24.341 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.339+0000 7f4984ff9640 10 monclient(hunting): dump: 2024-03-25T18:29:24.341 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:24.342 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:24.342 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:24.342 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:24.342 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:24.342 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:24.342 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:24.342 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:24.342 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:24.342 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:24.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.343+0000 7f4984ff9640 1 -- 172.21.15.3:0/571353817 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 294054680 0 0) 0x7f498813cb10 con 0x7f49880b44b0 2024-03-25T18:29:24.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.343+0000 7f4984ff9640 1 -- 172.21.15.3:0/571353817 >> v1:172.21.15.3:6790/0 conn(0x7f49880b1b00 legacy=0x7f4988129a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:24.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.343+0000 7f4984ff9640 1 -- 172.21.15.3:0/571353817 >> v1:172.21.15.3:6789/0 conn(0x7f49880b4e70 legacy=0x7f498812a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:24.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.343+0000 7f4984ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:29:24.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.343+0000 7f4984ff9640 1 monclient: found mon.b 2024-03-25T18:29:24.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.343+0000 7f4984ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:24.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.343+0000 7f4984ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:29:24.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.343+0000 7f4984ff9640 1 -- 172.21.15.3:0/571353817 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f498812acd0 con 0x7f49880b44b0 2024-03-25T18:29:24.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.343+0000 7f4984ff9640 10 monclient: _finish_auth 0 2024-03-25T18:29:24.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.343+0000 7f4984ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:29:24.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.343+0000 7f4984ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:24.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.343+0000 7f498e96b640 5 monclient: authenticate success, global_id 4393 2024-03-25T18:29:24.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.343+0000 7f498e96b640 10 monclient: _renew_subs 2024-03-25T18:29:24.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.343+0000 7f498e96b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:29:24.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.343+0000 7f498e96b640 1 -- 172.21.15.3:0/571353817 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f498813cdf0 con 0x7f49880b44b0 2024-03-25T18:29:24.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.343+0000 7f498e96b640 10 client.4393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:24.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.343+0000 7f498e96b640 10 monclient: _renew_subs 2024-03-25T18:29:24.346 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.343+0000 7f498e96b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:29:24.346 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.343+0000 7f498e96b640 1 -- 172.21.15.3:0/571353817 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f498813aa70 con 0x7f49880b44b0 2024-03-25T18:29:24.346 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.343+0000 7f498e96b640 1 librados: init done 2024-03-25T18:29:24.346 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.344+0000 7f498e96b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:24.346 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.344+0000 7f498e96b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:24.346 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.344+0000 7f498e96b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:29:24.347 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.344+0000 7f498e96b640 1 -- 172.21.15.3:0/571353817 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f498813ad20 con 0x7f49880b44b0 2024-03-25T18:29:24.638 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.636+0000 7f4984ff9640 1 -- 172.21.15.3:0/571353817 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f497c0047e0 con 0x7f49880b44b0 2024-03-25T18:29:24.638 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.636+0000 7f4984ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:24.638 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.636+0000 7f4984ff9640 1 -- 172.21.15.3:0/571353817 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f497c004af0 con 0x7f49880b44b0 2024-03-25T18:29:24.638 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.636+0000 7f4984ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:24.638 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.636+0000 7f4984ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:24.638 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.636+0000 7f4984ff9640 10 monclient: dump: 2024-03-25T18:29:24.638 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:24.638 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:24.638 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:24.638 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:24.639 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:24.639 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:24.639 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:24.639 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:24.639 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:24.639 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:24.639 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.636+0000 7f49867fc640 4 set_mon_vals no callback set 2024-03-25T18:29:24.640 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.637+0000 7f4984ff9640 1 -- 172.21.15.3:0/571353817 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f497c01aac0 con 0x7f49880b44b0 2024-03-25T18:29:24.640 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.638+0000 7f4987fff640 10 monclient: get_auth_request con 0x7f497406e110 auth_method 0 2024-03-25T18:29:24.640 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.638+0000 7f4984ff9640 10 client.4393.objecter ms_handle_connect 0x7f497406e110 2024-03-25T18:29:24.644 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.643+0000 7f4984ff9640 1 -- 172.21.15.3:0/571353817 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f497c054780 con 0x7f49880b44b0 2024-03-25T18:29:24.644 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.643+0000 7f4984ff9640 10 client.4393.objecter ms_dispatch 0x7f4988000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:24.644 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.643+0000 7f4984ff9640 3 client.4393.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:24.644 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.643+0000 7f4984ff9640 3 client.4393.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:24.644 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.643+0000 7f4984ff9640 20 client.4393.objecter dump_active .. 0 homeless 2024-03-25T18:29:24.644 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.643+0000 7f4984ff9640 1 -- 172.21.15.3:0/571353817 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f497c0500c0 con 0x7f49880b44b0 2024-03-25T18:29:24.644 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.643+0000 7f4984ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:24.644 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.643+0000 7f4984ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:24.900 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.898+0000 7f498e96b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:24.900 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.898+0000 7f498e96b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:24.900 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.898+0000 7f498e96b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:29:24.900 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:24.898+0000 7f498e96b640 1 -- 172.21.15.3:0/571353817 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f498804f730 con 0x7f49880b44b0 2024-03-25T18:29:25.011 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:25.010+0000 7f4984ff9640 1 -- 172.21.15.3:0/571353817 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f497c0521c0 con 0x7f49880b44b0 2024-03-25T18:29:25.011 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:25.010+0000 7f4984ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:25.011 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:25.010+0000 7f4984ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:25.011 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:29:25.012 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:25.014 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:25.012+0000 7f498e96b640 10 librados: watch_flush enter 2024-03-25T18:29:25.014 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:25.012+0000 7f498e96b640 10 librados: watch_flush exit 2024-03-25T18:29:25.014 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:25.012+0000 7f498e96b640 20 client.4393.objecter shutdown clearing up homeless session... 2024-03-25T18:29:25.014 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:25.012+0000 7f498e96b640 10 client.4393.objecter successfully canceled tick 2024-03-25T18:29:25.014 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:25.012+0000 7f498e96b640 1 -- 172.21.15.3:0/571353817 >> v1:172.21.15.3:6800/4195140966 conn(0x7f497406e110 legacy=0x7f49740705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:25.014 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:25.012+0000 7f498e96b640 10 monclient: shutdown 2024-03-25T18:29:25.015 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:25.012+0000 7f498e96b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:25.015 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:25.012+0000 7f498e96b640 1 -- 172.21.15.3:0/571353817 >> v1:172.21.15.16:6789/0 conn(0x7f49880b44b0 legacy=0x7f4988144f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:25.015 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:25.012+0000 7f498e96b640 1 -- 172.21.15.3:0/571353817 shutdown_connections 2024-03-25T18:29:25.015 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:25.012+0000 7f498e96b640 1 -- 172.21.15.3:0/571353817 >> 172.21.15.3:0/571353817 conn(0x7f49880617a0 msgr2=0x7f4988061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:25.015 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:25.013+0000 7f498e96b640 1 -- 172.21.15.3:0/571353817 shutdown_connections 2024-03-25T18:29:25.015 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:25.013+0000 7f498e96b640 1 -- 172.21.15.3:0/571353817 wait complete. 2024-03-25T18:29:25.015 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:25.013+0000 7f498e96b640 1 librados: shutdown 2024-03-25T18:29:25.025 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:31.832 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:31.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.932+0000 7f7a8bfc4640 10 monclient: get_monmap_and_config 2024-03-25T18:29:31.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.932+0000 7f7a8bfc4640 10 monclient: build_initial_monmap 2024-03-25T18:29:31.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.932+0000 7f7a8bfc4640 10 monclient: monmap: 2024-03-25T18:29:31.933 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:29:31.933 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:31.933 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:29:31.932968+0000 2024-03-25T18:29:31.933 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:29:31.932968+0000 2024-03-25T18:29:31.933 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:31.933 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:31.933 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:29:31.933 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:29:31.933 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:29:31.934 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:31.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.933+0000 7f7a8bfc4640 1 Processor -- start 2024-03-25T18:29:31.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.933+0000 7f7a8bfc4640 1 -- start start 2024-03-25T18:29:31.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.933+0000 7f7a8bfc4640 10 monclient: init 2024-03-25T18:29:31.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.934+0000 7f7a8bfc4640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:31.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.934+0000 7f7a8bfc4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:29:31.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.934+0000 7f7a8bfc4640 10 monclient(hunting): picked mon.noname-a con 0x7f7a840b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:29:31.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.934+0000 7f7a8bfc4640 10 monclient(hunting): picked mon.noname-b con 0x7f7a840b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:29:31.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.934+0000 7f7a8bfc4640 10 monclient(hunting): picked mon.noname-c con 0x7f7a8413c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:29:31.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.934+0000 7f7a8bfc4640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a840ab0e0 con 0x7f7a840b4ff0 2024-03-25T18:29:31.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.934+0000 7f7a8bfc4640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a840ab3c0 con 0x7f7a8413c480 2024-03-25T18:29:31.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.934+0000 7f7a8bfc4640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a840a9f30 con 0x7f7a840b4670 2024-03-25T18:29:31.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.934+0000 7f7a8bfc4640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:31.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.934+0000 7f7a8bfc4640 10 monclient(hunting): authenticate will time out at 7626.193675s 2024-03-25T18:29:31.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.958+0000 7f7a89d60640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f7a840b4ff0 0x7f7a8413bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:42412/0 (socket says 172.21.15.3:42412) 2024-03-25T18:29:31.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.958+0000 7f7a89d60640 1 -- 172.21.15.3:0/1479247674 learned_addr learned my addr 172.21.15.3:0/1479247674 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:29:31.979 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.979+0000 7f7a88d5e640 1 -- 172.21.15.3:0/1479247674 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4109931965 0 0) 0x7f7a840ab0e0 con 0x7f7a840b4ff0 2024-03-25T18:29:31.979 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.979+0000 7f7a88d5e640 10 monclient(hunting): _negotiate 2024-03-25T18:29:31.979 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.979+0000 7f7a88d5e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:31.979 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.979+0000 7f7a88d5e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:31.979 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.979+0000 7f7a88d5e640 10 monclient(hunting): my global_id is 4434 2024-03-25T18:29:31.980 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.979+0000 7f7a88d5e640 1 -- 172.21.15.3:0/1479247674 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a70003520 con 0x7f7a840b4ff0 2024-03-25T18:29:31.990 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.989+0000 7f7a88d5e640 1 -- 172.21.15.3:0/1479247674 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4029171723 0 0) 0x7f7a70003520 con 0x7f7a840b4ff0 2024-03-25T18:29:31.990 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.989+0000 7f7a88d5e640 1 -- 172.21.15.3:0/1479247674 >> v1:172.21.15.3:6790/0 conn(0x7f7a840b4670 legacy=0x7f7a841425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:29:31.990 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.989+0000 7f7a88d5e640 1 -- 172.21.15.3:0/1479247674 >> v1:172.21.15.16:6789/0 conn(0x7f7a8413c480 legacy=0x7f7a8414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:31.990 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.989+0000 7f7a88d5e640 10 monclient: _finish_hunting 0 2024-03-25T18:29:31.990 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.989+0000 7f7a88d5e640 1 monclient: found mon.noname-a 2024-03-25T18:29:31.990 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.989+0000 7f7a88d5e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:31.990 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.989+0000 7f7a88d5e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:29:31.990 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.989+0000 7f7a88d5e640 1 -- 172.21.15.3:0/1479247674 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a840a9b70 con 0x7f7a840b4ff0 2024-03-25T18:29:31.991 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.989+0000 7f7a88d5e640 10 monclient: _finish_auth 0 2024-03-25T18:29:31.991 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.989+0000 7f7a88d5e640 10 monclient: _check_auth_tickets 2024-03-25T18:29:31.991 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.989+0000 7f7a88d5e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:31.991 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.989+0000 7f7a88d5e640 1 -- 172.21.15.3:0/1479247674 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7a80001690 con 0x7f7a840b4ff0 2024-03-25T18:29:31.991 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.989+0000 7f7a8bfc4640 5 monclient: authenticate success, global_id 4434 2024-03-25T18:29:31.991 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.989+0000 7f7a8bfc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:31.991 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.989+0000 7f7a88d5e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:31.991 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.989+0000 7f7a88d5e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:31.991 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.989+0000 7f7a88d5e640 10 monclient: dump: 2024-03-25T18:29:31.991 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:31.991 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:31.991 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:31.991 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:31.991 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:31.992 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:31.992 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:31.992 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:31.992 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:31.992 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:31.992 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:31.989+0000 7f7a8bfc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:32.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.018+0000 7f7a88d5e640 1 -- 172.21.15.3:0/1479247674 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a80001860 con 0x7f7a840b4ff0 2024-03-25T18:29:32.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.018+0000 7f7a88d5e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:32.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.018+0000 7f7a88d5e640 1 -- 172.21.15.3:0/1479247674 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7a80001cb0 con 0x7f7a840b4ff0 2024-03-25T18:29:32.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.018+0000 7f7a88d5e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:32.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.018+0000 7f7a88d5e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:32.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.018+0000 7f7a88d5e640 10 monclient: dump: 2024-03-25T18:29:32.019 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:32.019 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:32.019 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:32.019 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:32.019 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:32.019 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:32.019 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:32.019 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:32.019 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:32.019 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:32.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.018+0000 7f7a8bfc4640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:32.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.018+0000 7f7a8bfc4640 4 set_mon_vals no callback set 2024-03-25T18:29:32.022 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.018+0000 7f7a8bfc4640 10 monclient: shutdown 2024-03-25T18:29:32.022 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.018+0000 7f7a8bfc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:32.022 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.018+0000 7f7a8bfc4640 1 -- 172.21.15.3:0/1479247674 >> v1:172.21.15.3:6789/0 conn(0x7f7a840b4ff0 legacy=0x7f7a8413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:32.022 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.019+0000 7f7a8bfc4640 1 -- 172.21.15.3:0/1479247674 shutdown_connections 2024-03-25T18:29:32.022 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.019+0000 7f7a8bfc4640 1 -- 172.21.15.3:0/1479247674 >> 172.21.15.3:0/1479247674 conn(0x7f7a840617a0 msgr2=0x7f7a84061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:32.022 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.019+0000 7f7a8bfc4640 1 -- 172.21.15.3:0/1479247674 shutdown_connections 2024-03-25T18:29:32.022 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.019+0000 7f7a8bfc4640 1 -- 172.21.15.3:0/1479247674 wait complete. 2024-03-25T18:29:32.022 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.020+0000 7f7a8bfc4640 10 monclient: build_initial_monmap 2024-03-25T18:29:32.022 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.020+0000 7f7a8bfc4640 10 monclient: monmap: 2024-03-25T18:29:32.023 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:29:32.023 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:32.023 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:29:32.023 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:29:32.023 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:32.023 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:29:32.023 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:29:32.023 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:29:32.023 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:29:32.023 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:32.023 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.020+0000 7f7a8bfc4640 1 librados: starting msgr at 2024-03-25T18:29:32.023 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.020+0000 7f7a8bfc4640 1 librados: starting objecter 2024-03-25T18:29:32.023 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.020+0000 7f7a8bfc4640 1 Processor -- start 2024-03-25T18:29:32.023 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.020+0000 7f7a8bfc4640 1 -- start start 2024-03-25T18:29:32.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.020+0000 7f7a8bfc4640 1 librados: setting wanted keys 2024-03-25T18:29:32.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.020+0000 7f7a8bfc4640 1 librados: calling monclient init 2024-03-25T18:29:32.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.020+0000 7f7a8bfc4640 10 monclient: init 2024-03-25T18:29:32.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.021+0000 7f7a8bfc4640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:32.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.021+0000 7f7a8bfc4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:29:32.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.021+0000 7f7a8bfc4640 10 monclient(hunting): picked mon.noname-b con 0x7f7a840b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:29:32.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.021+0000 7f7a8bfc4640 10 monclient(hunting): picked mon.noname-c con 0x7f7a840b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:29:32.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.021+0000 7f7a8bfc4640 10 monclient(hunting): picked mon.noname-a con 0x7f7a8413c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:29:32.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.021+0000 7f7a8bfc4640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a8413dcb0 con 0x7f7a8413c480 2024-03-25T18:29:32.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.021+0000 7f7a8bfc4640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a84145030 con 0x7f7a840b4670 2024-03-25T18:29:32.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.021+0000 7f7a8bfc4640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a8413df20 con 0x7f7a840b4ff0 2024-03-25T18:29:32.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.021+0000 7f7a8bfc4640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:32.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.021+0000 7f7a8bfc4640 10 monclient(hunting): authenticate will time out at 7626.280646s 2024-03-25T18:29:32.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.022+0000 7f7a8955f640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f7a840b4ff0 0x7f7a84143810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:36724/0 (socket says 172.21.15.3:36724) 2024-03-25T18:29:32.025 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.022+0000 7f7a8955f640 1 -- 172.21.15.3:0/572553032 learned_addr learned my addr 172.21.15.3:0/572553032 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:29:32.025 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.022+0000 7f7a7a7fc640 10 client.?.objecter ms_handle_connect 0x7f7a840b4670 2024-03-25T18:29:32.025 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.022+0000 7f7a7a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:32.025 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.023+0000 7f7a7a7fc640 10 client.?.objecter ms_handle_connect 0x7f7a840b4ff0 2024-03-25T18:29:32.025 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.023+0000 7f7a7a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:32.074 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.074+0000 7f7a7a7fc640 10 client.?.objecter ms_handle_connect 0x7f7a8413c480 2024-03-25T18:29:32.074 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.074+0000 7f7a7a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:32.097 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.096+0000 7f7a7a7fc640 1 -- 172.21.15.3:0/572553032 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386607746 0 0) 0x7f7a8413dcb0 con 0x7f7a8413c480 2024-03-25T18:29:32.097 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.097+0000 7f7a7a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:32.097 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.097+0000 7f7a7a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:32.097 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.097+0000 7f7a7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:32.097 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.097+0000 7f7a7a7fc640 10 monclient(hunting): my global_id is 4437 2024-03-25T18:29:32.097 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.097+0000 7f7a7a7fc640 1 -- 172.21.15.3:0/572553032 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a68004eb0 con 0x7f7a8413c480 2024-03-25T18:29:32.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.107+0000 7f7a7a7fc640 1 -- 172.21.15.3:0/572553032 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2847231378 0 0) 0x7f7a68004eb0 con 0x7f7a8413c480 2024-03-25T18:29:32.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.107+0000 7f7a7a7fc640 1 -- 172.21.15.3:0/572553032 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a8413dcb0 con 0x7f7a8413c480 2024-03-25T18:29:32.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.107+0000 7f7a7a7fc640 1 -- 172.21.15.3:0/572553032 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7a7c001b90 con 0x7f7a8413c480 2024-03-25T18:29:32.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.107+0000 7f7a7a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:32.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.107+0000 7f7a7a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:32.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.107+0000 7f7a7a7fc640 10 monclient(hunting): dump: 2024-03-25T18:29:32.108 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:32.108 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:32.108 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:32.108 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:32.108 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:32.108 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:32.108 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:32.109 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:32.109 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:32.109 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:32.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.107+0000 7f7a7a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:32.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.107+0000 7f7a7a7fc640 1 -- 172.21.15.3:0/572553032 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4181376800 0 0) 0x7f7a8413df20 con 0x7f7a840b4ff0 2024-03-25T18:29:32.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.107+0000 7f7a7a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:32.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.107+0000 7f7a7a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:32.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.107+0000 7f7a7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:32.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.107+0000 7f7a7a7fc640 10 monclient(hunting): my global_id is 4403 2024-03-25T18:29:32.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.107+0000 7f7a7a7fc640 1 -- 172.21.15.3:0/572553032 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a68004eb0 con 0x7f7a840b4ff0 2024-03-25T18:29:32.129 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.129+0000 7f7a7a7fc640 1 -- 172.21.15.3:0/572553032 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2943621050 0 0) 0x7f7a8413dcb0 con 0x7f7a8413c480 2024-03-25T18:29:32.129 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.129+0000 7f7a7a7fc640 1 -- 172.21.15.3:0/572553032 >> v1:172.21.15.3:6790/0 conn(0x7f7a840b4ff0 legacy=0x7f7a84143810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:32.129 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.129+0000 7f7a7a7fc640 1 -- 172.21.15.3:0/572553032 >> v1:172.21.15.16:6789/0 conn(0x7f7a840b4670 legacy=0x7f7a84144920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:32.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.129+0000 7f7a7a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:29:32.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.129+0000 7f7a7a7fc640 1 monclient: found mon.a 2024-03-25T18:29:32.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.129+0000 7f7a7a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:32.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.129+0000 7f7a7a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:29:32.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.129+0000 7f7a7a7fc640 1 -- 172.21.15.3:0/572553032 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a8413ce50 con 0x7f7a8413c480 2024-03-25T18:29:32.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.129+0000 7f7a7a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:29:32.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.129+0000 7f7a7a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:29:32.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.129+0000 7f7a7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:32.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.129+0000 7f7a8bfc4640 5 monclient: authenticate success, global_id 4437 2024-03-25T18:29:32.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.129+0000 7f7a8bfc4640 10 monclient: _renew_subs 2024-03-25T18:29:32.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.129+0000 7f7a8bfc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:29:32.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.129+0000 7f7a8bfc4640 1 -- 172.21.15.3:0/572553032 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a8413c0f0 con 0x7f7a8413c480 2024-03-25T18:29:32.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.129+0000 7f7a8bfc4640 10 client.4437.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:32.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.129+0000 7f7a8bfc4640 10 monclient: _renew_subs 2024-03-25T18:29:32.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.129+0000 7f7a8bfc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:29:32.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.129+0000 7f7a8bfc4640 1 -- 172.21.15.3:0/572553032 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a841467c0 con 0x7f7a8413c480 2024-03-25T18:29:32.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.129+0000 7f7a8bfc4640 1 librados: init done 2024-03-25T18:29:32.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.130+0000 7f7a8bfc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:32.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.130+0000 7f7a8bfc4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:32.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.130+0000 7f7a8bfc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:29:32.132 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.130+0000 7f7a8bfc4640 1 -- 172.21.15.3:0/572553032 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a8406dbd0 con 0x7f7a8413c480 2024-03-25T18:29:32.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.439+0000 7f7a7a7fc640 1 -- 172.21.15.3:0/572553032 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a7c0020c0 con 0x7f7a8413c480 2024-03-25T18:29:32.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.439+0000 7f7a7a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:32.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.439+0000 7f7a7a7fc640 1 -- 172.21.15.3:0/572553032 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7a7c003b30 con 0x7f7a8413c480 2024-03-25T18:29:32.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.439+0000 7f7a7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:32.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.439+0000 7f7a7a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:32.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.439+0000 7f7a7a7fc640 10 monclient: dump: 2024-03-25T18:29:32.440 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:32.440 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:32.440 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:32.440 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:32.440 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:32.440 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:32.440 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:32.440 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:32.440 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:32.440 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:32.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.439+0000 7f7a7bfff640 4 set_mon_vals no callback set 2024-03-25T18:29:32.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.440+0000 7f7a7a7fc640 1 -- 172.21.15.3:0/572553032 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f7a7c003d50 con 0x7f7a8413c480 2024-03-25T18:29:32.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.440+0000 7f7a89d60640 10 monclient: get_auth_request con 0x7f7a6806dd40 auth_method 0 2024-03-25T18:29:32.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.440+0000 7f7a7a7fc640 10 client.4437.objecter ms_handle_connect 0x7f7a6806dd40 2024-03-25T18:29:32.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.448+0000 7f7a7a7fc640 1 -- 172.21.15.3:0/572553032 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f7a7c053870 con 0x7f7a8413c480 2024-03-25T18:29:32.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.448+0000 7f7a7a7fc640 10 client.4437.objecter ms_dispatch 0x7f7a84000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:32.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.448+0000 7f7a7a7fc640 3 client.4437.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:32.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.448+0000 7f7a7a7fc640 3 client.4437.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:32.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.448+0000 7f7a7a7fc640 20 client.4437.objecter dump_active .. 0 homeless 2024-03-25T18:29:32.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.448+0000 7f7a7a7fc640 1 -- 172.21.15.3:0/572553032 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7a7c04f1b0 con 0x7f7a8413c480 2024-03-25T18:29:32.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.448+0000 7f7a7a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:32.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.448+0000 7f7a7a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:32.712 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.711+0000 7f7a8bfc4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:32.712 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.712+0000 7f7a8bfc4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:32.712 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.712+0000 7f7a8bfc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:29:32.712 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.712+0000 7f7a8bfc4640 1 -- 172.21.15.3:0/572553032 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a840688c0 con 0x7f7a8413c480 2024-03-25T18:29:32.741 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.741+0000 7f7a7a7fc640 1 -- 172.21.15.3:0/572553032 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f7a7c0512b0 con 0x7f7a8413c480 2024-03-25T18:29:32.741 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.741+0000 7f7a7a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:32.741 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.741+0000 7f7a7a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:32.742 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:29:32.742 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:32.744 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.743+0000 7f7a8bfc4640 10 librados: watch_flush enter 2024-03-25T18:29:32.744 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.743+0000 7f7a8bfc4640 10 librados: watch_flush exit 2024-03-25T18:29:32.744 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.743+0000 7f7a8bfc4640 20 client.4437.objecter shutdown clearing up homeless session... 2024-03-25T18:29:32.744 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.743+0000 7f7a8bfc4640 10 client.4437.objecter successfully canceled tick 2024-03-25T18:29:32.744 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.743+0000 7f7a8bfc4640 1 -- 172.21.15.3:0/572553032 >> v1:172.21.15.3:6800/4195140966 conn(0x7f7a6806dd40 legacy=0x7f7a68070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:32.744 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.743+0000 7f7a8bfc4640 10 monclient: shutdown 2024-03-25T18:29:32.744 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.743+0000 7f7a8bfc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:32.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.743+0000 7f7a8bfc4640 1 -- 172.21.15.3:0/572553032 >> v1:172.21.15.3:6789/0 conn(0x7f7a8413c480 legacy=0x7f7a841e71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:32.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.744+0000 7f7a8bfc4640 1 -- 172.21.15.3:0/572553032 shutdown_connections 2024-03-25T18:29:32.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.744+0000 7f7a8bfc4640 1 -- 172.21.15.3:0/572553032 >> 172.21.15.3:0/572553032 conn(0x7f7a840617a0 msgr2=0x7f7a840b8690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:32.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.744+0000 7f7a8bfc4640 1 -- 172.21.15.3:0/572553032 shutdown_connections 2024-03-25T18:29:32.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.744+0000 7f7a8bfc4640 1 -- 172.21.15.3:0/572553032 wait complete. 2024-03-25T18:29:32.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:32.744+0000 7f7a8bfc4640 1 librados: shutdown 2024-03-25T18:29:32.756 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:39.564 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:39.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.677+0000 7f3ccf972640 10 monclient: get_monmap_and_config 2024-03-25T18:29:39.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.677+0000 7f3ccf972640 10 monclient: build_initial_monmap 2024-03-25T18:29:39.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.677+0000 7f3ccf972640 10 monclient: monmap: 2024-03-25T18:29:39.679 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:29:39.679 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:39.679 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:29:39.679006+0000 2024-03-25T18:29:39.679 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:29:39.679006+0000 2024-03-25T18:29:39.679 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:39.679 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:39.679 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:29:39.680 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:29:39.680 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:29:39.680 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:39.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.678+0000 7f3ccf972640 1 Processor -- start 2024-03-25T18:29:39.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.678+0000 7f3ccf972640 1 -- start start 2024-03-25T18:29:39.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.678+0000 7f3ccf972640 10 monclient: init 2024-03-25T18:29:39.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.679+0000 7f3ccf972640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:39.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.679+0000 7f3ccf972640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:29:39.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.679+0000 7f3ccf972640 10 monclient(hunting): picked mon.noname-a con 0x7f3cc80b4e70 addr v1:172.21.15.3:6789/0 2024-03-25T18:29:39.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.679+0000 7f3ccf972640 10 monclient(hunting): picked mon.noname-b con 0x7f3cc80b44b0 addr v1:172.21.15.3:6790/0 2024-03-25T18:29:39.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.679+0000 7f3ccf972640 10 monclient(hunting): picked mon.noname-c con 0x7f3cc80b1b00 addr v1:172.21.15.16:6789/0 2024-03-25T18:29:39.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.679+0000 7f3ccf972640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cc80a96f0 con 0x7f3cc80b4e70 2024-03-25T18:29:39.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.679+0000 7f3ccf972640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cc80ab0e0 con 0x7f3cc80b1b00 2024-03-25T18:29:39.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.679+0000 7f3ccf972640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cc8145f70 con 0x7f3cc80b44b0 2024-03-25T18:29:39.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.679+0000 7f3ccf972640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:39.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.679+0000 7f3ccf972640 10 monclient(hunting): authenticate will time out at 7633.939793s 2024-03-25T18:29:39.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.681+0000 7f3cccf0d640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f3cc80b44b0 0x7f3cc8145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:54762/0 (socket says 172.21.15.3:54762) 2024-03-25T18:29:39.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.681+0000 7f3cccf0d640 1 -- 172.21.15.3:0/1969065798 learned_addr learned my addr 172.21.15.3:0/1969065798 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:29:39.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.741+0000 7f3cbffff640 1 -- 172.21.15.3:0/1969065798 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 437905827 0 0) 0x7f3cc80a96f0 con 0x7f3cc80b4e70 2024-03-25T18:29:39.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.741+0000 7f3cbffff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:39.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.741+0000 7f3cbffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:39.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.741+0000 7f3cbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:39.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.741+0000 7f3cbffff640 10 monclient(hunting): my global_id is 4440 2024-03-25T18:29:39.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.741+0000 7f3cbffff640 1 -- 172.21.15.3:0/1969065798 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cb4003520 con 0x7f3cc80b4e70 2024-03-25T18:29:39.756 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.754+0000 7f3cbffff640 1 -- 172.21.15.3:0/1969065798 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1525873442 0 0) 0x7f3cb4003520 con 0x7f3cc80b4e70 2024-03-25T18:29:39.756 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.754+0000 7f3cbffff640 1 -- 172.21.15.3:0/1969065798 >> v1:172.21.15.3:6790/0 conn(0x7f3cc80b44b0 legacy=0x7f3cc8145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:39.756 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.754+0000 7f3cbffff640 1 -- 172.21.15.3:0/1969065798 >> v1:172.21.15.16:6789/0 conn(0x7f3cc80b1b00 legacy=0x7f3cc814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:39.756 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.754+0000 7f3cbffff640 10 monclient: _finish_hunting 0 2024-03-25T18:29:39.756 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.754+0000 7f3cbffff640 1 monclient: found mon.noname-a 2024-03-25T18:29:39.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.754+0000 7f3cbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:39.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.754+0000 7f3cbffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:29:39.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.754+0000 7f3cbffff640 1 -- 172.21.15.3:0/1969065798 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc80a9b80 con 0x7f3cc80b4e70 2024-03-25T18:29:39.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.754+0000 7f3cbffff640 10 monclient: _finish_auth 0 2024-03-25T18:29:39.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.754+0000 7f3cbffff640 10 monclient: _check_auth_tickets 2024-03-25T18:29:39.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.754+0000 7f3cbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:39.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.754+0000 7f3cbffff640 1 -- 172.21.15.3:0/1969065798 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f3cc4001690 con 0x7f3cc80b4e70 2024-03-25T18:29:39.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.754+0000 7f3ccf972640 5 monclient: authenticate success, global_id 4440 2024-03-25T18:29:39.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.754+0000 7f3ccf972640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:39.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.754+0000 7f3cbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:39.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.754+0000 7f3cbffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:39.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.754+0000 7f3cbffff640 10 monclient: dump: 2024-03-25T18:29:39.758 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:39.759 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:39.759 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:39.759 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:39.759 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:39.760 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:39.760 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:39.760 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:39.760 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:39.760 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:39.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.754+0000 7f3ccf972640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:39.772 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.771+0000 7f3cbffff640 1 -- 172.21.15.3:0/1969065798 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3cc40017f0 con 0x7f3cc80b4e70 2024-03-25T18:29:39.772 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.771+0000 7f3cbffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:39.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.771+0000 7f3cbffff640 1 -- 172.21.15.3:0/1969065798 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f3cc4001c00 con 0x7f3cc80b4e70 2024-03-25T18:29:39.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.771+0000 7f3cbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:39.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.771+0000 7f3cbffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:39.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.771+0000 7f3cbffff640 10 monclient: dump: 2024-03-25T18:29:39.773 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:39.773 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:39.773 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:39.773 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:39.773 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:39.773 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:39.773 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:39.773 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:39.773 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:39.773 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:39.774 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.771+0000 7f3ccf972640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:39.774 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.771+0000 7f3ccf972640 4 set_mon_vals no callback set 2024-03-25T18:29:39.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.772+0000 7f3ccf972640 10 monclient: shutdown 2024-03-25T18:29:39.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.772+0000 7f3ccf972640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:39.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.772+0000 7f3ccf972640 1 -- 172.21.15.3:0/1969065798 >> v1:172.21.15.3:6789/0 conn(0x7f3cc80b4e70 legacy=0x7f3cc813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:39.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.772+0000 7f3ccf972640 1 -- 172.21.15.3:0/1969065798 shutdown_connections 2024-03-25T18:29:39.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.772+0000 7f3ccf972640 1 -- 172.21.15.3:0/1969065798 >> 172.21.15.3:0/1969065798 conn(0x7f3cc80617a0 msgr2=0x7f3cc8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:39.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.772+0000 7f3ccf972640 1 -- 172.21.15.3:0/1969065798 shutdown_connections 2024-03-25T18:29:39.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.772+0000 7f3ccf972640 1 -- 172.21.15.3:0/1969065798 wait complete. 2024-03-25T18:29:39.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.773+0000 7f3ccf972640 10 monclient: build_initial_monmap 2024-03-25T18:29:39.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.773+0000 7f3ccf972640 10 monclient: monmap: 2024-03-25T18:29:39.776 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:29:39.776 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:39.777 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:29:39.777 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:29:39.777 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:39.777 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:29:39.777 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:29:39.777 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:29:39.777 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:29:39.777 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:39.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.773+0000 7f3ccf972640 1 librados: starting msgr at 2024-03-25T18:29:39.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.773+0000 7f3ccf972640 1 librados: starting objecter 2024-03-25T18:29:39.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.773+0000 7f3ccf972640 1 Processor -- start 2024-03-25T18:29:39.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.773+0000 7f3ccf972640 1 -- start start 2024-03-25T18:29:39.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.773+0000 7f3ccf972640 1 librados: setting wanted keys 2024-03-25T18:29:39.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.773+0000 7f3ccf972640 1 librados: calling monclient init 2024-03-25T18:29:39.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.773+0000 7f3ccf972640 10 monclient: init 2024-03-25T18:29:39.778 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.774+0000 7f3ccf972640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:39.778 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.774+0000 7f3ccf972640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:29:39.778 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.774+0000 7f3ccf972640 10 monclient(hunting): picked mon.noname-a con 0x7f3cc80b1b00 addr v1:172.21.15.3:6789/0 2024-03-25T18:29:39.778 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.774+0000 7f3ccf972640 10 monclient(hunting): picked mon.noname-c con 0x7f3cc80b44b0 addr v1:172.21.15.3:6790/0 2024-03-25T18:29:39.778 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.774+0000 7f3ccf972640 10 monclient(hunting): picked mon.noname-b con 0x7f3cc80b4e70 addr v1:172.21.15.16:6789/0 2024-03-25T18:29:39.778 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.774+0000 7f3ccf972640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cc81451c0 con 0x7f3cc80b1b00 2024-03-25T18:29:39.778 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.774+0000 7f3ccf972640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cc813b6e0 con 0x7f3cc80b4e70 2024-03-25T18:29:39.778 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.774+0000 7f3ccf972640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cc813cb50 con 0x7f3cc80b44b0 2024-03-25T18:29:39.778 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.774+0000 7f3ccf972640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:39.778 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.774+0000 7f3ccf972640 10 monclient(hunting): authenticate will time out at 7634.034500s 2024-03-25T18:29:39.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.785+0000 7f3ccd70e640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f3cc80b1b00 0x7f3cc813aa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:35044/0 (socket says 172.21.15.3:35044) 2024-03-25T18:29:39.787 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.785+0000 7f3ccd70e640 1 -- 172.21.15.3:0/52628648 learned_addr learned my addr 172.21.15.3:0/52628648 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:29:39.789 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.787+0000 7f3cbdffb640 10 client.?.objecter ms_handle_connect 0x7f3cc80b1b00 2024-03-25T18:29:39.789 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.787+0000 7f3cbdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:39.799 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.798+0000 7f3cbdffb640 1 -- 172.21.15.3:0/52628648 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 785450870 0 0) 0x7f3cc81451c0 con 0x7f3cc80b1b00 2024-03-25T18:29:39.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.798+0000 7f3cbdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:39.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.798+0000 7f3cbdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:39.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.798+0000 7f3cbdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:39.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.798+0000 7f3cbdffb640 10 monclient(hunting): my global_id is 4443 2024-03-25T18:29:39.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.798+0000 7f3cbdffb640 1 -- 172.21.15.3:0/52628648 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cac004eb0 con 0x7f3cc80b1b00 2024-03-25T18:29:39.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.806+0000 7f3cbdffb640 10 client.?.objecter ms_handle_connect 0x7f3cc80b4e70 2024-03-25T18:29:39.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.806+0000 7f3cbdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:39.821 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.820+0000 7f3cbdffb640 1 -- 172.21.15.3:0/52628648 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2956609539 0 0) 0x7f3cac004eb0 con 0x7f3cc80b1b00 2024-03-25T18:29:39.821 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.820+0000 7f3cbdffb640 1 -- 172.21.15.3:0/52628648 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3cc81451c0 con 0x7f3cc80b1b00 2024-03-25T18:29:39.821 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.820+0000 7f3cbdffb640 1 -- 172.21.15.3:0/52628648 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f3cc4001690 con 0x7f3cc80b1b00 2024-03-25T18:29:39.821 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.820+0000 7f3cbdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:39.821 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.820+0000 7f3cbdffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:39.821 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.820+0000 7f3cbdffb640 10 monclient(hunting): dump: 2024-03-25T18:29:39.822 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:39.822 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:39.822 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:39.822 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:39.822 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:39.822 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:39.822 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:39.822 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:39.822 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:39.822 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:39.822 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.820+0000 7f3cbdffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:39.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.827+0000 7f3cbdffb640 10 client.?.objecter ms_handle_connect 0x7f3cc80b44b0 2024-03-25T18:29:39.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.827+0000 7f3cbdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:39.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.830+0000 7f3cbdffb640 1 -- 172.21.15.3:0/52628648 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 639759631 0 0) 0x7f3cc81451c0 con 0x7f3cc80b1b00 2024-03-25T18:29:39.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.830+0000 7f3cbdffb640 1 -- 172.21.15.3:0/52628648 >> v1:172.21.15.3:6790/0 conn(0x7f3cc80b44b0 legacy=0x7f3cc8143b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:39.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.830+0000 7f3cbdffb640 1 -- 172.21.15.3:0/52628648 >> v1:172.21.15.16:6789/0 conn(0x7f3cc80b4e70 legacy=0x7f3cc8142790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:39.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.830+0000 7f3cbdffb640 10 monclient: _finish_hunting 0 2024-03-25T18:29:39.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.830+0000 7f3cbdffb640 1 monclient: found mon.a 2024-03-25T18:29:39.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.830+0000 7f3cbdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:39.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.830+0000 7f3cbdffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:29:39.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.830+0000 7f3cbdffb640 1 -- 172.21.15.3:0/52628648 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc8139f50 con 0x7f3cc80b1b00 2024-03-25T18:29:39.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.830+0000 7f3cbdffb640 10 monclient: _finish_auth 0 2024-03-25T18:29:39.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.830+0000 7f3cbdffb640 10 monclient: _check_auth_tickets 2024-03-25T18:29:39.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.830+0000 7f3cbdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:39.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.830+0000 7f3ccf972640 5 monclient: authenticate success, global_id 4443 2024-03-25T18:29:39.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.831+0000 7f3ccf972640 10 monclient: _renew_subs 2024-03-25T18:29:39.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.831+0000 7f3ccf972640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:29:39.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.831+0000 7f3ccf972640 1 -- 172.21.15.3:0/52628648 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cc813bb70 con 0x7f3cc80b1b00 2024-03-25T18:29:39.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.831+0000 7f3ccf972640 10 client.4443.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:39.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.831+0000 7f3ccf972640 10 monclient: _renew_subs 2024-03-25T18:29:39.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.831+0000 7f3ccf972640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:29:39.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.831+0000 7f3ccf972640 1 -- 172.21.15.3:0/52628648 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cc8144210 con 0x7f3cc80b1b00 2024-03-25T18:29:39.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.831+0000 7f3ccf972640 1 librados: init done 2024-03-25T18:29:39.834 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.832+0000 7f3ccf972640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:39.834 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.832+0000 7f3ccf972640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:39.834 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.832+0000 7f3ccf972640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:29:39.834 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:39.832+0000 7f3ccf972640 1 -- 172.21.15.3:0/52628648 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cc813b450 con 0x7f3cc80b1b00 2024-03-25T18:29:40.152 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.150+0000 7f3cbdffb640 1 -- 172.21.15.3:0/52628648 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3cc40038a0 con 0x7f3cc80b1b00 2024-03-25T18:29:40.152 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.151+0000 7f3cbdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:40.152 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.151+0000 7f3cbdffb640 1 -- 172.21.15.3:0/52628648 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f3cc4003c70 con 0x7f3cc80b1b00 2024-03-25T18:29:40.152 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.151+0000 7f3cbdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:40.152 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.151+0000 7f3cbdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:40.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.151+0000 7f3cbdffb640 10 monclient: dump: 2024-03-25T18:29:40.153 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:40.153 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:40.153 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:40.153 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:40.153 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:40.153 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:40.153 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:40.153 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:40.153 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:40.153 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:40.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.151+0000 7f3cbf7fe640 4 set_mon_vals no callback set 2024-03-25T18:29:40.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.151+0000 7f3cbdffb640 1 -- 172.21.15.3:0/52628648 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f3cc4003e90 con 0x7f3cc80b1b00 2024-03-25T18:29:40.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.152+0000 7f3cccf0d640 10 monclient: get_auth_request con 0x7f3cac06dd90 auth_method 0 2024-03-25T18:29:40.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.152+0000 7f3cbdffb640 10 client.4443.objecter ms_handle_connect 0x7f3cac06dd90 2024-03-25T18:29:40.159 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.157+0000 7f3cbdffb640 1 -- 172.21.15.3:0/52628648 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f3cc4054980 con 0x7f3cc80b1b00 2024-03-25T18:29:40.159 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.157+0000 7f3cbdffb640 10 client.4443.objecter ms_dispatch 0x7f3cc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:40.159 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.157+0000 7f3cbdffb640 3 client.4443.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:40.159 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.157+0000 7f3cbdffb640 3 client.4443.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:40.159 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.157+0000 7f3cbdffb640 20 client.4443.objecter dump_active .. 0 homeless 2024-03-25T18:29:40.159 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.157+0000 7f3cbdffb640 1 -- 172.21.15.3:0/52628648 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3cc4050490 con 0x7f3cc80b1b00 2024-03-25T18:29:40.159 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.157+0000 7f3cbdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:40.159 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.157+0000 7f3cbdffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:40.417 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.416+0000 7f3ccf972640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:40.417 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.416+0000 7f3ccf972640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:40.418 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.416+0000 7f3ccf972640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:29:40.418 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.416+0000 7f3ccf972640 1 -- 172.21.15.3:0/52628648 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3cc804f730 con 0x7f3cc80b1b00 2024-03-25T18:29:40.437 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.436+0000 7f3cbdffb640 1 -- 172.21.15.3:0/52628648 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f3cc40523c0 con 0x7f3cc80b1b00 2024-03-25T18:29:40.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.436+0000 7f3cbdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:40.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.436+0000 7f3cbdffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:40.438 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:29:40.438 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:40.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.438+0000 7f3ccf972640 10 librados: watch_flush enter 2024-03-25T18:29:40.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.439+0000 7f3ccf972640 10 librados: watch_flush exit 2024-03-25T18:29:40.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.439+0000 7f3ccf972640 20 client.4443.objecter shutdown clearing up homeless session... 2024-03-25T18:29:40.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.439+0000 7f3ccf972640 10 client.4443.objecter successfully canceled tick 2024-03-25T18:29:40.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.439+0000 7f3ccf972640 1 -- 172.21.15.3:0/52628648 >> v1:172.21.15.3:6800/4195140966 conn(0x7f3cac06dd90 legacy=0x7f3cac070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:40.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.439+0000 7f3ccf972640 10 monclient: shutdown 2024-03-25T18:29:40.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.439+0000 7f3ccf972640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:40.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.439+0000 7f3ccf972640 1 -- 172.21.15.3:0/52628648 >> v1:172.21.15.3:6789/0 conn(0x7f3cc80b1b00 legacy=0x7f3cc813aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:40.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.439+0000 7f3ccf972640 1 -- 172.21.15.3:0/52628648 shutdown_connections 2024-03-25T18:29:40.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.439+0000 7f3ccf972640 1 -- 172.21.15.3:0/52628648 >> 172.21.15.3:0/52628648 conn(0x7f3cc80617a0 msgr2=0x7f3cc80b6690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:40.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.439+0000 7f3ccf972640 1 -- 172.21.15.3:0/52628648 shutdown_connections 2024-03-25T18:29:40.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.439+0000 7f3ccf972640 1 -- 172.21.15.3:0/52628648 wait complete. 2024-03-25T18:29:40.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:40.439+0000 7f3ccf972640 1 librados: shutdown 2024-03-25T18:29:40.452 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:41.757 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:04:55.486 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:41.804 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:04:55.496 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:41.806 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:04:55.498 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:41.808 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:04:55.537 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:41.858 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:04:55.550 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:41.858 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:04:55.587 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:41.861 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:04:55.590 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:41.876 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:04:55.605 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:41.903 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:04:55.632 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:47.259 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:47.375 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.373+0000 7f46a9cf6640 10 monclient: get_monmap_and_config 2024-03-25T18:29:47.376 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.373+0000 7f46a9cf6640 10 monclient: build_initial_monmap 2024-03-25T18:29:47.376 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.373+0000 7f46a9cf6640 10 monclient: monmap: 2024-03-25T18:29:47.376 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:29:47.376 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:47.376 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:29:47.374872+0000 2024-03-25T18:29:47.376 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:29:47.374872+0000 2024-03-25T18:29:47.376 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:47.376 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:47.376 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:29:47.376 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:29:47.376 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:29:47.377 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:47.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.374+0000 7f46a9cf6640 1 Processor -- start 2024-03-25T18:29:47.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.374+0000 7f46a9cf6640 1 -- start start 2024-03-25T18:29:47.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.374+0000 7f46a9cf6640 10 monclient: init 2024-03-25T18:29:47.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.375+0000 7f46a9cf6640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:47.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.375+0000 7f46a9cf6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:29:47.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.375+0000 7f46a9cf6640 10 monclient(hunting): picked mon.noname-c con 0x7f46a40b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:29:47.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.375+0000 7f46a9cf6640 10 monclient(hunting): picked mon.noname-b con 0x7f46a40b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:29:47.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.375+0000 7f46a9cf6640 10 monclient(hunting): picked mon.noname-a con 0x7f46a413c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:29:47.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.375+0000 7f46a9cf6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a40ab0e0 con 0x7f46a413c480 2024-03-25T18:29:47.379 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.375+0000 7f46a9cf6640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a40ab3c0 con 0x7f46a40b4ff0 2024-03-25T18:29:47.379 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.375+0000 7f46a9cf6640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a40a9f30 con 0x7f46a40b4670 2024-03-25T18:29:47.379 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.375+0000 7f46a9cf6640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:47.379 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.375+0000 7f46a9cf6640 10 monclient(hunting): authenticate will time out at 7641.636154s 2024-03-25T18:29:47.388 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.387+0000 7f46a37fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f46a40b4ff0 0x7f46a413bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:43540/0 (socket says 172.21.15.3:43540) 2024-03-25T18:29:47.388 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.387+0000 7f46a37fe640 1 -- 172.21.15.3:0/453095787 learned_addr learned my addr 172.21.15.3:0/453095787 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:29:47.414 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.412+0000 7f46a2ffd640 1 -- 172.21.15.3:0/453095787 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1238485343 0 0) 0x7f46a40ab0e0 con 0x7f46a413c480 2024-03-25T18:29:47.414 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.412+0000 7f46a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:47.414 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.412+0000 7f46a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:47.414 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.412+0000 7f46a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:47.414 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.412+0000 7f46a2ffd640 10 monclient(hunting): my global_id is 4446 2024-03-25T18:29:47.414 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.412+0000 7f46a2ffd640 1 -- 172.21.15.3:0/453095787 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4690003520 con 0x7f46a413c480 2024-03-25T18:29:47.418 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.417+0000 7f46a2ffd640 1 -- 172.21.15.3:0/453095787 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2015635786 0 0) 0x7f46a40a9f30 con 0x7f46a40b4670 2024-03-25T18:29:47.418 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.417+0000 7f46a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:47.419 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.417+0000 7f46a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:47.419 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.417+0000 7f46a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:47.419 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.417+0000 7f46a2ffd640 10 monclient(hunting): my global_id is 4412 2024-03-25T18:29:47.419 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.417+0000 7f46a2ffd640 1 -- 172.21.15.3:0/453095787 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46a40ab0e0 con 0x7f46a40b4670 2024-03-25T18:29:47.419 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.417+0000 7f46a2ffd640 1 -- 172.21.15.3:0/453095787 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3490366291 0 0) 0x7f46a40ab3c0 con 0x7f46a40b4ff0 2024-03-25T18:29:47.419 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.417+0000 7f46a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:47.419 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.417+0000 7f46a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:47.419 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.417+0000 7f46a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:47.419 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.417+0000 7f46a2ffd640 10 monclient(hunting): my global_id is 4402 2024-03-25T18:29:47.419 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.417+0000 7f46a2ffd640 1 -- 172.21.15.3:0/453095787 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46a40a9f30 con 0x7f46a40b4ff0 2024-03-25T18:29:47.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.426+0000 7f46a2ffd640 1 -- 172.21.15.3:0/453095787 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1926536657 0 0) 0x7f46a40a9f30 con 0x7f46a40b4ff0 2024-03-25T18:29:47.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.426+0000 7f46a2ffd640 1 -- 172.21.15.3:0/453095787 >> v1:172.21.15.3:6790/0 conn(0x7f46a40b4670 legacy=0x7f46a41425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:47.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.426+0000 7f46a2ffd640 1 -- 172.21.15.3:0/453095787 >> v1:172.21.15.3:6789/0 conn(0x7f46a413c480 legacy=0x7f46a414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:47.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.426+0000 7f46a2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:29:47.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.426+0000 7f46a2ffd640 1 monclient: found mon.noname-c 2024-03-25T18:29:47.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.426+0000 7f46a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:47.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.426+0000 7f46a2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:29:47.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.426+0000 7f46a2ffd640 1 -- 172.21.15.3:0/453095787 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a40a9b70 con 0x7f46a40b4ff0 2024-03-25T18:29:47.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.426+0000 7f46a2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:29:47.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.426+0000 7f46a2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:29:47.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.426+0000 7f46a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:47.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.426+0000 7f46a2ffd640 1 -- 172.21.15.3:0/453095787 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f46940015b0 con 0x7f46a40b4ff0 2024-03-25T18:29:47.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.426+0000 7f46a9cf6640 5 monclient: authenticate success, global_id 4402 2024-03-25T18:29:47.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.426+0000 7f46a9cf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:47.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.426+0000 7f46a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:47.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.426+0000 7f46a2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:47.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.426+0000 7f46a2ffd640 10 monclient: dump: 2024-03-25T18:29:47.429 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:47.429 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:47.429 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:47.429 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:47.429 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:47.429 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:47.429 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:47.429 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:47.429 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:47.429 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:47.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.426+0000 7f46a9cf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:47.443 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.442+0000 7f46a2ffd640 1 -- 172.21.15.3:0/453095787 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4694001710 con 0x7f46a40b4ff0 2024-03-25T18:29:47.443 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.442+0000 7f46a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:47.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.442+0000 7f46a2ffd640 1 -- 172.21.15.3:0/453095787 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4694001b60 con 0x7f46a40b4ff0 2024-03-25T18:29:47.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.442+0000 7f46a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:47.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.442+0000 7f46a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:47.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.442+0000 7f46a2ffd640 10 monclient: dump: 2024-03-25T18:29:47.444 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:47.444 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:47.444 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:47.444 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:47.445 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:47.445 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:47.445 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:47.445 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:47.445 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:47.445 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:47.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.442+0000 7f46a9cf6640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:47.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.442+0000 7f46a9cf6640 4 set_mon_vals no callback set 2024-03-25T18:29:47.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.443+0000 7f46a9cf6640 10 monclient: shutdown 2024-03-25T18:29:47.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.443+0000 7f46a9cf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:47.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.443+0000 7f46a9cf6640 1 -- 172.21.15.3:0/453095787 >> v1:172.21.15.16:6789/0 conn(0x7f46a40b4ff0 legacy=0x7f46a413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:47.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.443+0000 7f46a9cf6640 1 -- 172.21.15.3:0/453095787 shutdown_connections 2024-03-25T18:29:47.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.443+0000 7f46a9cf6640 1 -- 172.21.15.3:0/453095787 >> 172.21.15.3:0/453095787 conn(0x7f46a40617a0 msgr2=0x7f46a4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:47.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.443+0000 7f46a9cf6640 1 -- 172.21.15.3:0/453095787 shutdown_connections 2024-03-25T18:29:47.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.443+0000 7f46a9cf6640 1 -- 172.21.15.3:0/453095787 wait complete. 2024-03-25T18:29:47.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.444+0000 7f46a9cf6640 10 monclient: build_initial_monmap 2024-03-25T18:29:47.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.444+0000 7f46a9cf6640 10 monclient: monmap: 2024-03-25T18:29:47.449 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:29:47.449 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:47.449 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:29:47.449 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:29:47.450 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:47.450 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:29:47.450 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:29:47.450 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:29:47.450 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:29:47.450 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:47.450 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.445+0000 7f46a9cf6640 1 librados: starting msgr at 2024-03-25T18:29:47.450 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.445+0000 7f46a9cf6640 1 librados: starting objecter 2024-03-25T18:29:47.450 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.445+0000 7f46a9cf6640 1 Processor -- start 2024-03-25T18:29:47.451 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.445+0000 7f46a9cf6640 1 -- start start 2024-03-25T18:29:47.451 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.445+0000 7f46a9cf6640 1 librados: setting wanted keys 2024-03-25T18:29:47.451 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.445+0000 7f46a9cf6640 1 librados: calling monclient init 2024-03-25T18:29:47.451 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.445+0000 7f46a9cf6640 10 monclient: init 2024-03-25T18:29:47.451 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.446+0000 7f46a9cf6640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:47.451 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.446+0000 7f46a9cf6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:29:47.451 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.446+0000 7f46a9cf6640 10 monclient(hunting): picked mon.noname-a con 0x7f46a40b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:29:47.451 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.446+0000 7f46a9cf6640 10 monclient(hunting): picked mon.noname-c con 0x7f46a40b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:29:47.451 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.446+0000 7f46a9cf6640 10 monclient(hunting): picked mon.noname-b con 0x7f46a413c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:29:47.451 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.446+0000 7f46a9cf6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a40ac2b0 con 0x7f46a40b4670 2024-03-25T18:29:47.451 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.446+0000 7f46a9cf6640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a41d7760 con 0x7f46a413c480 2024-03-25T18:29:47.451 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.446+0000 7f46a9cf6640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a41d8390 con 0x7f46a40b4ff0 2024-03-25T18:29:47.451 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.446+0000 7f46a9cf6640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:47.451 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.446+0000 7f46a9cf6640 10 monclient(hunting): authenticate will time out at 7641.706843s 2024-03-25T18:29:47.466 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.465+0000 7f469bfff640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f46a40b4ff0 0x7f46a41d47d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:59300/0 (socket says 172.21.15.3:59300) 2024-03-25T18:29:47.466 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.465+0000 7f469bfff640 1 -- 172.21.15.3:0/1165810111 learned_addr learned my addr 172.21.15.3:0/1165810111 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:29:47.467 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.466+0000 7f46a0ff9640 10 client.?.objecter ms_handle_connect 0x7f46a40b4ff0 2024-03-25T18:29:47.468 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.466+0000 7f46a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:47.469 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.468+0000 7f46a0ff9640 10 client.?.objecter ms_handle_connect 0x7f46a413c480 2024-03-25T18:29:47.469 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.468+0000 7f46a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:47.470 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.468+0000 7f46a0ff9640 10 client.?.objecter ms_handle_connect 0x7f46a40b4670 2024-03-25T18:29:47.470 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.468+0000 7f46a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:47.481 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.480+0000 7f46a0ff9640 1 -- 172.21.15.3:0/1165810111 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 283495241 0 0) 0x7f46a40ac2b0 con 0x7f46a40b4670 2024-03-25T18:29:47.482 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.480+0000 7f46a0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:29:47.482 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.480+0000 7f46a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:47.482 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.480+0000 7f46a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:47.482 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.480+0000 7f46a0ff9640 10 monclient(hunting): my global_id is 4449 2024-03-25T18:29:47.482 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.480+0000 7f46a0ff9640 1 -- 172.21.15.3:0/1165810111 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4684004eb0 con 0x7f46a40b4670 2024-03-25T18:29:47.483 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.480+0000 7f46a0ff9640 1 -- 172.21.15.3:0/1165810111 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2157042778 0 0) 0x7f46a41d7760 con 0x7f46a413c480 2024-03-25T18:29:47.483 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.480+0000 7f46a0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:29:47.483 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.480+0000 7f46a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:47.483 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.480+0000 7f46a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:47.483 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.480+0000 7f46a0ff9640 10 monclient(hunting): my global_id is 4405 2024-03-25T18:29:47.483 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.480+0000 7f46a0ff9640 1 -- 172.21.15.3:0/1165810111 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46a40ac2b0 con 0x7f46a413c480 2024-03-25T18:29:47.484 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.482+0000 7f46a0ff9640 1 -- 172.21.15.3:0/1165810111 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2857567163 0 0) 0x7f46a41d8390 con 0x7f46a40b4ff0 2024-03-25T18:29:47.484 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.482+0000 7f46a0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:29:47.485 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.482+0000 7f46a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:47.485 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.482+0000 7f46a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:47.485 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.482+0000 7f46a0ff9640 10 monclient(hunting): my global_id is 4415 2024-03-25T18:29:47.485 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.482+0000 7f46a0ff9640 1 -- 172.21.15.3:0/1165810111 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46a41d7760 con 0x7f46a40b4ff0 2024-03-25T18:29:47.495 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.493+0000 7f46a0ff9640 1 -- 172.21.15.3:0/1165810111 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2933206276 0 0) 0x7f46a40ac2b0 con 0x7f46a413c480 2024-03-25T18:29:47.495 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.493+0000 7f46a0ff9640 1 -- 172.21.15.3:0/1165810111 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46a41d8390 con 0x7f46a413c480 2024-03-25T18:29:47.496 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.493+0000 7f46a0ff9640 1 -- 172.21.15.3:0/1165810111 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f469c001ab0 con 0x7f46a413c480 2024-03-25T18:29:47.496 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.493+0000 7f46a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:47.496 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.493+0000 7f46a0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:47.496 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.493+0000 7f46a0ff9640 10 monclient(hunting): dump: 2024-03-25T18:29:47.496 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:47.496 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:47.496 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:47.496 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:47.496 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:47.496 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:47.496 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:47.496 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:47.496 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:47.496 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:47.497 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.493+0000 7f46a0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:47.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.501+0000 7f46a0ff9640 1 -- 172.21.15.3:0/1165810111 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3873072261 0 0) 0x7f4684004eb0 con 0x7f46a40b4670 2024-03-25T18:29:47.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.501+0000 7f46a0ff9640 1 -- 172.21.15.3:0/1165810111 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46a40ac2b0 con 0x7f46a40b4670 2024-03-25T18:29:47.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.501+0000 7f46a0ff9640 1 -- 172.21.15.3:0/1165810111 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4694001a20 con 0x7f46a40b4670 2024-03-25T18:29:47.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.501+0000 7f46a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:47.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.501+0000 7f46a0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:47.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.501+0000 7f46a0ff9640 10 monclient(hunting): dump: 2024-03-25T18:29:47.503 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:47.503 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:47.503 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:47.503 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:47.503 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:47.503 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:47.503 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:47.503 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:47.503 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:47.503 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:47.508 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.507+0000 7f46a0ff9640 1 -- 172.21.15.3:0/1165810111 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3128106607 0 0) 0x7f46a41d7760 con 0x7f46a40b4ff0 2024-03-25T18:29:47.508 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.507+0000 7f46a0ff9640 1 -- 172.21.15.3:0/1165810111 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4684004eb0 con 0x7f46a40b4ff0 2024-03-25T18:29:47.508 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.507+0000 7f46a0ff9640 1 -- 172.21.15.3:0/1165810111 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f468c0019b0 con 0x7f46a40b4ff0 2024-03-25T18:29:47.508 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.507+0000 7f46a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:47.508 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.507+0000 7f46a0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:47.509 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.507+0000 7f46a0ff9640 10 monclient(hunting): dump: 2024-03-25T18:29:47.509 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:47.509 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:47.509 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:47.509 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:47.509 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:47.509 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:47.509 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:47.509 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:47.509 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:47.509 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:47.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.520+0000 7f46a0ff9640 1 -- 172.21.15.3:0/1165810111 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2087490740 0 0) 0x7f46a41d8390 con 0x7f46a413c480 2024-03-25T18:29:47.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.520+0000 7f46a0ff9640 1 -- 172.21.15.3:0/1165810111 >> v1:172.21.15.3:6790/0 conn(0x7f46a40b4ff0 legacy=0x7f46a41d47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:47.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.520+0000 7f46a0ff9640 1 -- 172.21.15.3:0/1165810111 >> v1:172.21.15.3:6789/0 conn(0x7f46a40b4670 legacy=0x7f46a41d7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:47.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.520+0000 7f46a0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:29:47.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.520+0000 7f46a0ff9640 1 monclient: found mon.b 2024-03-25T18:29:47.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.520+0000 7f46a0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:47.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.520+0000 7f46a0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:29:47.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.520+0000 7f46a0ff9640 1 -- 172.21.15.3:0/1165810111 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a41d4ee0 con 0x7f46a413c480 2024-03-25T18:29:47.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.520+0000 7f46a0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:29:47.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.520+0000 7f46a0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:29:47.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.520+0000 7f46a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:47.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.520+0000 7f46a9cf6640 5 monclient: authenticate success, global_id 4405 2024-03-25T18:29:47.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.520+0000 7f46a9cf6640 10 monclient: _renew_subs 2024-03-25T18:29:47.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.520+0000 7f46a9cf6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:29:47.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.520+0000 7f46a9cf6640 1 -- 172.21.15.3:0/1165810111 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46a41d5190 con 0x7f46a413c480 2024-03-25T18:29:47.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.520+0000 7f46a9cf6640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:47.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.520+0000 7f46a9cf6640 10 monclient: _renew_subs 2024-03-25T18:29:47.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.520+0000 7f46a9cf6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:29:47.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.520+0000 7f46a9cf6640 1 -- 172.21.15.3:0/1165810111 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46a41d42a0 con 0x7f46a413c480 2024-03-25T18:29:47.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.520+0000 7f46a9cf6640 1 librados: init done 2024-03-25T18:29:47.524 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.521+0000 7f46a9cf6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:47.524 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.521+0000 7f46a9cf6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:47.524 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.521+0000 7f46a9cf6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:29:47.524 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.521+0000 7f46a9cf6640 1 -- 172.21.15.3:0/1165810111 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46a41d75c0 con 0x7f46a413c480 2024-03-25T18:29:47.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.741+0000 7f46a0ff9640 1 -- 172.21.15.3:0/1165810111 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f469c001f80 con 0x7f46a413c480 2024-03-25T18:29:47.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.741+0000 7f46a0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:47.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.741+0000 7f46a27fc640 4 set_mon_vals no callback set 2024-03-25T18:29:47.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.829+0000 7f46a0ff9640 1 -- 172.21.15.3:0/1165810111 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f469c003970 con 0x7f46a413c480 2024-03-25T18:29:47.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.829+0000 7f46a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:47.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.829+0000 7f46a0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:47.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.829+0000 7f46a0ff9640 10 monclient: dump: 2024-03-25T18:29:47.831 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:47.831 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:47.831 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:47.831 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:47.831 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:47.831 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:47.831 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:47.831 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:47.832 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:47.832 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:47.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.830+0000 7f46a0ff9640 1 -- 172.21.15.3:0/1165810111 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f469c019940 con 0x7f46a413c480 2024-03-25T18:29:47.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.831+0000 7f46a37fe640 10 monclient: get_auth_request con 0x7f468406e0b0 auth_method 0 2024-03-25T18:29:47.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.831+0000 7f46a0ff9640 10 client.4405.objecter ms_handle_connect 0x7f468406e0b0 2024-03-25T18:29:47.834 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.833+0000 7f46a0ff9640 1 -- 172.21.15.3:0/1165810111 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f469c053620 con 0x7f46a413c480 2024-03-25T18:29:47.834 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.833+0000 7f46a0ff9640 10 client.4405.objecter ms_dispatch 0x7f46a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:47.834 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.833+0000 7f46a0ff9640 3 client.4405.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:47.834 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.833+0000 7f46a0ff9640 3 client.4405.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:47.834 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.833+0000 7f46a0ff9640 20 client.4405.objecter dump_active .. 0 homeless 2024-03-25T18:29:47.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.836+0000 7f46a0ff9640 1 -- 172.21.15.3:0/1165810111 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f469c04ef60 con 0x7f46a413c480 2024-03-25T18:29:47.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.836+0000 7f46a0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:47.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:47.836+0000 7f46a0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:48.057 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:48.056+0000 7f46a9cf6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:48.057 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:48.056+0000 7f46a9cf6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:48.057 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:48.056+0000 7f46a9cf6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:29:48.057 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:48.056+0000 7f46a9cf6640 1 -- 172.21.15.3:0/1165810111 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46a40ac0b0 con 0x7f46a413c480 2024-03-25T18:29:48.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:48.120+0000 7f46a0ff9640 1 -- 172.21.15.3:0/1165810111 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f469c051060 con 0x7f46a413c480 2024-03-25T18:29:48.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:48.120+0000 7f46a0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:48.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:48.120+0000 7f46a0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:48.122 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:29:48.122 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:48.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:48.122+0000 7f46a9cf6640 10 librados: watch_flush enter 2024-03-25T18:29:48.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:48.122+0000 7f46a9cf6640 10 librados: watch_flush exit 2024-03-25T18:29:48.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:48.122+0000 7f46a9cf6640 20 client.4405.objecter shutdown clearing up homeless session... 2024-03-25T18:29:48.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:48.122+0000 7f46a9cf6640 10 client.4405.objecter successfully canceled tick 2024-03-25T18:29:48.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:48.122+0000 7f46a9cf6640 1 -- 172.21.15.3:0/1165810111 >> v1:172.21.15.3:6800/4195140966 conn(0x7f468406e0b0 legacy=0x7f4684070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:48.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:48.122+0000 7f46a9cf6640 10 monclient: shutdown 2024-03-25T18:29:48.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:48.122+0000 7f46a9cf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:48.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:48.122+0000 7f46a9cf6640 1 -- 172.21.15.3:0/1165810111 >> v1:172.21.15.16:6789/0 conn(0x7f46a413c480 legacy=0x7f46a413b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:48.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:48.122+0000 7f46a9cf6640 1 -- 172.21.15.3:0/1165810111 shutdown_connections 2024-03-25T18:29:48.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:48.122+0000 7f46a9cf6640 1 -- 172.21.15.3:0/1165810111 >> 172.21.15.3:0/1165810111 conn(0x7f46a40617a0 msgr2=0x7f46a4140670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:48.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:48.122+0000 7f46a9cf6640 1 -- 172.21.15.3:0/1165810111 shutdown_connections 2024-03-25T18:29:48.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:48.122+0000 7f46a9cf6640 1 -- 172.21.15.3:0/1165810111 wait complete. 2024-03-25T18:29:48.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:48.122+0000 7f46a9cf6640 1 librados: shutdown 2024-03-25T18:29:48.136 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:54.944 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:55.057 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.056+0000 7f923e7af640 10 monclient: get_monmap_and_config 2024-03-25T18:29:55.057 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.056+0000 7f923e7af640 10 monclient: build_initial_monmap 2024-03-25T18:29:55.057 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.056+0000 7f923e7af640 10 monclient: monmap: 2024-03-25T18:29:55.057 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:29:55.058 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:55.058 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:29:55.057489+0000 2024-03-25T18:29:55.058 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:29:55.057489+0000 2024-03-25T18:29:55.058 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:55.058 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:55.058 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:29:55.058 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:29:55.058 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:29:55.058 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:55.059 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.057+0000 7f923e7af640 1 Processor -- start 2024-03-25T18:29:55.059 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.057+0000 7f923e7af640 1 -- start start 2024-03-25T18:29:55.059 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.057+0000 7f923e7af640 10 monclient: init 2024-03-25T18:29:55.059 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.058+0000 7f923e7af640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:55.059 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.058+0000 7f923e7af640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:29:55.059 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.058+0000 7f923e7af640 10 monclient(hunting): picked mon.noname-c con 0x7f92380b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:29:55.059 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.058+0000 7f923e7af640 10 monclient(hunting): picked mon.noname-a con 0x7f92380b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:29:55.059 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.058+0000 7f923e7af640 10 monclient(hunting): picked mon.noname-b con 0x7f923813c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:29:55.060 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.058+0000 7f923e7af640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92380ab0e0 con 0x7f92380b4670 2024-03-25T18:29:55.060 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.058+0000 7f923e7af640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92380ab3c0 con 0x7f92380b4ff0 2024-03-25T18:29:55.060 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.058+0000 7f923e7af640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92380a9f30 con 0x7f923813c480 2024-03-25T18:29:55.060 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.058+0000 7f923e7af640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:55.060 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.058+0000 7f923e7af640 10 monclient(hunting): authenticate will time out at 7649.318591s 2024-03-25T18:29:55.063 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.062+0000 7f9237fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f92380b4ff0 0x7f923813bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:59526/0 (socket says 172.21.15.3:59526) 2024-03-25T18:29:55.063 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.062+0000 7f9237fff640 1 -- 172.21.15.3:0/3530682914 learned_addr learned my addr 172.21.15.3:0/3530682914 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:29:55.069 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.068+0000 7f9236ffd640 1 -- 172.21.15.3:0/3530682914 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 688260743 0 0) 0x7f92380ab3c0 con 0x7f92380b4ff0 2024-03-25T18:29:55.070 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.068+0000 7f9236ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:55.070 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.068+0000 7f9236ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:55.070 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.068+0000 7f9236ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:55.070 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.068+0000 7f9236ffd640 10 monclient(hunting): my global_id is 4408 2024-03-25T18:29:55.070 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.068+0000 7f9236ffd640 1 -- 172.21.15.3:0/3530682914 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9220003520 con 0x7f92380b4ff0 2024-03-25T18:29:55.079 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.078+0000 7f9236ffd640 1 -- 172.21.15.3:0/3530682914 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1443584168 0 0) 0x7f9220003520 con 0x7f92380b4ff0 2024-03-25T18:29:55.079 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.078+0000 7f9236ffd640 1 -- 172.21.15.3:0/3530682914 >> v1:172.21.15.3:6790/0 conn(0x7f923813c480 legacy=0x7f923814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:55.079 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.078+0000 7f9236ffd640 1 -- 172.21.15.3:0/3530682914 >> v1:172.21.15.3:6789/0 conn(0x7f92380b4670 legacy=0x7f92381425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:29:55.080 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.078+0000 7f9236ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:29:55.080 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.078+0000 7f9236ffd640 1 monclient: found mon.noname-c 2024-03-25T18:29:55.080 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.078+0000 7f9236ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:55.080 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.078+0000 7f9236ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:29:55.080 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.078+0000 7f9236ffd640 1 -- 172.21.15.3:0/3530682914 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92380a9b70 con 0x7f92380b4ff0 2024-03-25T18:29:55.080 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.078+0000 7f9236ffd640 10 monclient: _finish_auth 0 2024-03-25T18:29:55.080 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.078+0000 7f9236ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:29:55.080 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.078+0000 7f9236ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:55.080 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.078+0000 7f9236ffd640 1 -- 172.21.15.3:0/3530682914 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f9228001670 con 0x7f92380b4ff0 2024-03-25T18:29:55.080 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.078+0000 7f9236ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:55.080 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.078+0000 7f9236ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:55.080 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.078+0000 7f9236ffd640 10 monclient: dump: 2024-03-25T18:29:55.080 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:55.080 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:55.081 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:55.081 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:55.081 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:55.081 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:55.081 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:55.081 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:55.081 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:55.081 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:55.081 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.078+0000 7f923e7af640 5 monclient: authenticate success, global_id 4408 2024-03-25T18:29:55.081 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.078+0000 7f923e7af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:55.086 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.085+0000 7f9236ffd640 1 -- 172.21.15.3:0/3530682914 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9228001840 con 0x7f92380b4ff0 2024-03-25T18:29:55.086 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.085+0000 7f9236ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:55.086 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.085+0000 7f9236ffd640 1 -- 172.21.15.3:0/3530682914 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f9228001c50 con 0x7f92380b4ff0 2024-03-25T18:29:55.086 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.085+0000 7f923e7af640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:55.086 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.085+0000 7f923e7af640 4 set_mon_vals no callback set 2024-03-25T18:29:55.086 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.085+0000 7f9236ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:55.086 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.085+0000 7f9236ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:55.087 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.085+0000 7f9236ffd640 10 monclient: dump: 2024-03-25T18:29:55.087 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:55.087 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:55.087 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:55.087 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:55.087 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:55.087 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:55.087 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:55.087 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:55.087 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:55.087 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:55.090 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.086+0000 7f923e7af640 10 monclient: shutdown 2024-03-25T18:29:55.090 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.086+0000 7f923e7af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:55.090 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.086+0000 7f923e7af640 1 -- 172.21.15.3:0/3530682914 >> v1:172.21.15.16:6789/0 conn(0x7f92380b4ff0 legacy=0x7f923813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:55.090 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.086+0000 7f923e7af640 1 -- 172.21.15.3:0/3530682914 shutdown_connections 2024-03-25T18:29:55.090 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.086+0000 7f923e7af640 1 -- 172.21.15.3:0/3530682914 >> 172.21.15.3:0/3530682914 conn(0x7f92380617a0 msgr2=0x7f9238061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:55.090 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.086+0000 7f923e7af640 1 -- 172.21.15.3:0/3530682914 shutdown_connections 2024-03-25T18:29:55.090 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.086+0000 7f923e7af640 1 -- 172.21.15.3:0/3530682914 wait complete. 2024-03-25T18:29:55.090 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.086+0000 7f923e7af640 10 monclient: build_initial_monmap 2024-03-25T18:29:55.090 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.086+0000 7f923e7af640 10 monclient: monmap: 2024-03-25T18:29:55.091 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:29:55.091 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:55.091 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:29:55.091 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:29:55.091 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:55.091 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:29:55.091 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:29:55.091 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:29:55.091 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:29:55.091 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:55.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.087+0000 7f923e7af640 1 librados: starting msgr at 2024-03-25T18:29:55.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.087+0000 7f923e7af640 1 librados: starting objecter 2024-03-25T18:29:55.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.087+0000 7f923e7af640 1 Processor -- start 2024-03-25T18:29:55.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.087+0000 7f923e7af640 1 -- start start 2024-03-25T18:29:55.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.087+0000 7f923e7af640 1 librados: setting wanted keys 2024-03-25T18:29:55.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.087+0000 7f923e7af640 1 librados: calling monclient init 2024-03-25T18:29:55.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.087+0000 7f923e7af640 10 monclient: init 2024-03-25T18:29:55.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.088+0000 7f923e7af640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:55.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.088+0000 7f923e7af640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:29:55.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.088+0000 7f923e7af640 10 monclient(hunting): picked mon.noname-b con 0x7f92380b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:29:55.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.088+0000 7f923e7af640 10 monclient(hunting): picked mon.noname-c con 0x7f92380b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:29:55.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.088+0000 7f923e7af640 10 monclient(hunting): picked mon.noname-a con 0x7f923813c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:29:55.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.088+0000 7f923e7af640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9238130bd0 con 0x7f923813c480 2024-03-25T18:29:55.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.088+0000 7f923e7af640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92381449a0 con 0x7f92380b4670 2024-03-25T18:29:55.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.088+0000 7f923e7af640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9238144c80 con 0x7f92380b4ff0 2024-03-25T18:29:55.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.088+0000 7f923e7af640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:55.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.088+0000 7f923e7af640 10 monclient(hunting): authenticate will time out at 7649.348627s 2024-03-25T18:29:55.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.109+0000 7f923cd4c640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f923813c480 0x7f92381304c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:38812/0 (socket says 172.21.15.3:38812) 2024-03-25T18:29:55.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.109+0000 7f923cd4c640 1 -- 172.21.15.3:0/598329502 learned_addr learned my addr 172.21.15.3:0/598329502 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:29:55.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.110+0000 7f9234ff9640 10 client.?.objecter ms_handle_connect 0x7f923813c480 2024-03-25T18:29:55.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.110+0000 7f9234ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:55.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.113+0000 7f9234ff9640 10 client.?.objecter ms_handle_connect 0x7f92380b4ff0 2024-03-25T18:29:55.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.113+0000 7f9234ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:55.115 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.115+0000 7f9234ff9640 10 client.?.objecter ms_handle_connect 0x7f92380b4670 2024-03-25T18:29:55.116 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.115+0000 7f9234ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:55.128 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.127+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 276455992 0 0) 0x7f92381449a0 con 0x7f92380b4670 2024-03-25T18:29:55.128 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.127+0000 7f9234ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:29:55.129 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.127+0000 7f9234ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:55.129 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.127+0000 7f9234ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:55.129 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.127+0000 7f9234ff9640 10 monclient(hunting): my global_id is 4411 2024-03-25T18:29:55.129 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.127+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9224004eb0 con 0x7f92380b4670 2024-03-25T18:29:55.129 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.128+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1752453023 0 0) 0x7f9238130bd0 con 0x7f923813c480 2024-03-25T18:29:55.129 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.128+0000 7f9234ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:29:55.129 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.128+0000 7f9234ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:55.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.128+0000 7f9234ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:55.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.128+0000 7f9234ff9640 10 monclient(hunting): my global_id is 4452 2024-03-25T18:29:55.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.128+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92381449a0 con 0x7f923813c480 2024-03-25T18:29:55.136 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.135+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3308426318 0 0) 0x7f9238144c80 con 0x7f92380b4ff0 2024-03-25T18:29:55.136 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.135+0000 7f9234ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:29:55.137 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.135+0000 7f9234ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:55.137 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.135+0000 7f9234ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:55.137 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.135+0000 7f9234ff9640 10 monclient(hunting): my global_id is 4421 2024-03-25T18:29:55.137 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.135+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9238130bd0 con 0x7f92380b4ff0 2024-03-25T18:29:55.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.152+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2441227427 0 0) 0x7f9224004eb0 con 0x7f92380b4670 2024-03-25T18:29:55.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.152+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9238144c80 con 0x7f92380b4670 2024-03-25T18:29:55.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.152+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f9228001b50 con 0x7f92380b4670 2024-03-25T18:29:55.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.152+0000 7f9234ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:55.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.152+0000 7f9234ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:55.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.152+0000 7f9234ff9640 10 monclient(hunting): dump: 2024-03-25T18:29:55.154 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:55.154 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:55.154 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:55.154 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:55.155 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:55.155 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:55.155 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:55.155 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:55.155 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:55.155 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:55.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.152+0000 7f9234ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:55.176 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.175+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1122940660 0 0) 0x7f9238130bd0 con 0x7f92380b4ff0 2024-03-25T18:29:55.176 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.175+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9224004eb0 con 0x7f92380b4ff0 2024-03-25T18:29:55.176 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.175+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f922c001510 con 0x7f92380b4ff0 2024-03-25T18:29:55.176 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.175+0000 7f9234ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:55.176 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.175+0000 7f9234ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:55.176 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.175+0000 7f9234ff9640 10 monclient(hunting): dump: 2024-03-25T18:29:55.176 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:55.176 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:55.176 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:55.176 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:55.176 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:55.177 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:55.177 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:55.177 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:55.177 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:55.177 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:55.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.175+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4214756499 0 0) 0x7f92381449a0 con 0x7f923813c480 2024-03-25T18:29:55.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.175+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9238130bd0 con 0x7f923813c480 2024-03-25T18:29:55.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.175+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f9230001c10 con 0x7f923813c480 2024-03-25T18:29:55.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.175+0000 7f9234ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:55.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.175+0000 7f9234ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:55.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.175+0000 7f9234ff9640 10 monclient(hunting): dump: 2024-03-25T18:29:55.179 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:55.179 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:55.179 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:55.179 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:55.179 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:55.179 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:55.179 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:55.179 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:55.180 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:55.180 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:55.187 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.185+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1679964449 0 0) 0x7f9238144c80 con 0x7f92380b4670 2024-03-25T18:29:55.189 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.185+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 >> v1:172.21.15.3:6790/0 conn(0x7f92380b4ff0 legacy=0x7f923812fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:55.189 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.185+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 >> v1:172.21.15.3:6789/0 conn(0x7f923813c480 legacy=0x7f92381304c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:55.189 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.185+0000 7f9234ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:29:55.189 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.185+0000 7f9234ff9640 1 monclient: found mon.b 2024-03-25T18:29:55.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.185+0000 7f9234ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:55.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.185+0000 7f9234ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:29:55.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.185+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9238144f60 con 0x7f92380b4670 2024-03-25T18:29:55.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.185+0000 7f9234ff9640 10 monclient: _finish_auth 0 2024-03-25T18:29:55.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.185+0000 7f9234ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:29:55.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.185+0000 7f9234ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:55.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.185+0000 7f923e7af640 5 monclient: authenticate success, global_id 4411 2024-03-25T18:29:55.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.185+0000 7f923e7af640 10 monclient: _renew_subs 2024-03-25T18:29:55.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.185+0000 7f923e7af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:29:55.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.185+0000 7f923e7af640 1 -- 172.21.15.3:0/598329502 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9238145660 con 0x7f92380b4670 2024-03-25T18:29:55.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.185+0000 7f923e7af640 10 client.4411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:55.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.185+0000 7f923e7af640 10 monclient: _renew_subs 2024-03-25T18:29:55.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.185+0000 7f923e7af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:29:55.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.185+0000 7f923e7af640 1 -- 172.21.15.3:0/598329502 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92381465e0 con 0x7f92380b4670 2024-03-25T18:29:55.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.185+0000 7f923e7af640 1 librados: init done 2024-03-25T18:29:55.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.186+0000 7f923e7af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:55.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.186+0000 7f923e7af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:55.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.186+0000 7f923e7af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:29:55.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.186+0000 7f923e7af640 1 -- 172.21.15.3:0/598329502 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f923813ff50 con 0x7f92380b4670 2024-03-25T18:29:55.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.202+0000 7f9237fff640 1 -- 172.21.15.3:0/598329502 >> v1:172.21.15.16:6789/0 conn(0x7f92380b4670 legacy=0x7f9238145ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T18:29:55.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.202+0000 7f9237fff640 1 -- 172.21.15.3:0/598329502 >> v1:172.21.15.16:6789/0 conn(0x7f92380b4670 legacy=0x7f9238145ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:29:55.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.202+0000 7f9237fff640 1 --1- 172.21.15.3:0/598329502 >> v1:172.21.15.16:6789/0 conn(0x7f92380b4670 0x7f9238145ed0 :-1 s=OPENED pgs=32051 cs=1 l=1).handle_message read tag failed 2024-03-25T18:29:55.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.202+0000 7f9237fff640 1 --1- 172.21.15.3:0/598329502 >> v1:172.21.15.16:6789/0 conn(0x7f92380b4670 0x7f9238145ed0 :-1 s=OPENED pgs=32051 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:29:55.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.202+0000 7f9234ff9640 10 monclient: ms_handle_reset current mon v1:172.21.15.16:6789/0 2024-03-25T18:29:55.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.202+0000 7f9234ff9640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:55.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.202+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 >> v1:172.21.15.16:6789/0 conn(0x7f92380b4670 legacy=0x7f9238145ed0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:29:55.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.202+0000 7f9234ff9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:29:55.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.202+0000 7f9234ff9640 10 monclient(hunting): picked mon.b con 0x7f9224006ae0 addr v1:172.21.15.16:6789/0 2024-03-25T18:29:55.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.202+0000 7f9234ff9640 10 monclient(hunting): picked mon.c con 0x7f92380b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:29:55.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.202+0000 7f9234ff9640 10 monclient(hunting): picked mon.a con 0x7f923813c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:29:55.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.202+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9238144c80 con 0x7f923813c480 2024-03-25T18:29:55.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.202+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f92381449a0 con 0x7f9224006ae0 2024-03-25T18:29:55.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.202+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9224014020 con 0x7f92380b4ff0 2024-03-25T18:29:55.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.202+0000 7f9234ff9640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:55.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.217+0000 7f9234ff9640 10 client.4411.objecter ms_handle_connect 0x7f923813c480 2024-03-25T18:29:55.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.217+0000 7f9234ff9640 10 client.4411.objecter resend_mon_ops 2024-03-25T18:29:55.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.220+0000 7f9234ff9640 10 client.4411.objecter ms_handle_connect 0x7f92380b4ff0 2024-03-25T18:29:55.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.220+0000 7f9234ff9640 10 client.4411.objecter resend_mon_ops 2024-03-25T18:29:55.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.228+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991570079 0 0) 0x7f9238144c80 con 0x7f923813c480 2024-03-25T18:29:55.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.228+0000 7f9234ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:29:55.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.228+0000 7f9234ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:55.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.228+0000 7f9234ff9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T18:29:55.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.228+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 --> v1:172.21.15.3:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f92240141b0 con 0x7f923813c480 2024-03-25T18:29:55.235 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.234+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3342030075 0 0) 0x7f9224014020 con 0x7f92380b4ff0 2024-03-25T18:29:55.235 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.234+0000 7f9234ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:29:55.235 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.234+0000 7f9234ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:55.236 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.234+0000 7f9234ff9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T18:29:55.236 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.234+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 --> v1:172.21.15.3:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f9238144c80 con 0x7f92380b4ff0 2024-03-25T18:29:55.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.237+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 1538778994 0 0) 0x7f92240141b0 con 0x7f923813c480 2024-03-25T18:29:55.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.237+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 >> v1:172.21.15.3:6790/0 conn(0x7f92380b4ff0 legacy=0x7f922400ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:55.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.237+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 >> v1:172.21.15.16:6789/0 conn(0x7f9224006ae0 legacy=0x7f92240092f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:55.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.237+0000 7f9234ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:29:55.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.237+0000 7f9234ff9640 1 monclient: found mon.a 2024-03-25T18:29:55.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.237+0000 7f9234ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:55.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.237+0000 7f9234ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:29:55.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.237+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f9224015400 con 0x7f923813c480 2024-03-25T18:29:55.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.237+0000 7f9234ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:55.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.237+0000 7f9234ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:29:55.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.237+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92381449a0 con 0x7f923813c480 2024-03-25T18:29:55.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.237+0000 7f9234ff9640 10 monclient: _finish_auth 0 2024-03-25T18:29:55.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.237+0000 7f9234ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:29:55.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.237+0000 7f9234ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:55.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.237+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f9230003990 con 0x7f923813c480 2024-03-25T18:29:55.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.237+0000 7f9234ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:55.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.237+0000 7f9234ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:55.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.237+0000 7f9234ff9640 10 monclient: dump: 2024-03-25T18:29:55.240 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:29:55.240 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:29:55.240 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:55.240 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:29:55.240 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:55.240 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:29:55.240 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:29:55.240 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:29:55.240 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:29:55.240 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:29:55.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.545+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9230003af0 con 0x7f923813c480 2024-03-25T18:29:55.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.545+0000 7f9234ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:55.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.545+0000 7f92367fc640 4 set_mon_vals no callback set 2024-03-25T18:29:55.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.545+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f9230003ce0 con 0x7f923813c480 2024-03-25T18:29:55.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.546+0000 7f9237fff640 10 monclient: get_auth_request con 0x7f92380b4670 auth_method 0 2024-03-25T18:29:55.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.546+0000 7f9234ff9640 10 client.4411.objecter ms_handle_connect 0x7f92380b4670 2024-03-25T18:29:55.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.551+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 <== mon.0 v1:172.21.15.3:6789/0 6 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f9230054ad0 con 0x7f923813c480 2024-03-25T18:29:55.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.551+0000 7f9234ff9640 10 client.4411.objecter ms_dispatch 0x7f9238000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:55.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.551+0000 7f9234ff9640 3 client.4411.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:55.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.551+0000 7f9234ff9640 3 client.4411.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:55.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.551+0000 7f9234ff9640 20 client.4411.objecter dump_active .. 0 homeless 2024-03-25T18:29:55.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.551+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9230050410 con 0x7f923813c480 2024-03-25T18:29:55.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.552+0000 7f9234ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:55.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.552+0000 7f9234ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:55.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.804+0000 7f923e7af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:55.806 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.804+0000 7f923e7af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:55.806 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.804+0000 7f923e7af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:29:55.806 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.804+0000 7f923e7af640 1 -- 172.21.15.3:0/598329502 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92380a9450 con 0x7f923813c480 2024-03-25T18:29:55.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.834+0000 7f9234ff9640 1 -- 172.21.15.3:0/598329502 <== mon.0 v1:172.21.15.3:6789/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f9230052510 con 0x7f923813c480 2024-03-25T18:29:55.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.835+0000 7f9234ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:55.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.835+0000 7f9234ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:55.836 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:29:55.836 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:55.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.837+0000 7f923e7af640 10 librados: watch_flush enter 2024-03-25T18:29:55.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.837+0000 7f923e7af640 10 librados: watch_flush exit 2024-03-25T18:29:55.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.837+0000 7f923e7af640 20 client.4411.objecter shutdown clearing up homeless session... 2024-03-25T18:29:55.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.837+0000 7f923e7af640 10 client.4411.objecter successfully canceled tick 2024-03-25T18:29:55.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.837+0000 7f923e7af640 1 -- 172.21.15.3:0/598329502 >> v1:172.21.15.3:6800/4195140966 conn(0x7f92380b4670 legacy=0x7f922407bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:55.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.837+0000 7f923e7af640 10 monclient: shutdown 2024-03-25T18:29:55.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.837+0000 7f923e7af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:55.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.837+0000 7f923e7af640 1 -- 172.21.15.3:0/598329502 >> v1:172.21.15.3:6789/0 conn(0x7f923813c480 legacy=0x7f9224010410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:55.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.837+0000 7f923e7af640 1 -- 172.21.15.3:0/598329502 shutdown_connections 2024-03-25T18:29:55.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.837+0000 7f923e7af640 1 -- 172.21.15.3:0/598329502 >> 172.21.15.3:0/598329502 conn(0x7f92380617a0 msgr2=0x7f92381407f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:55.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.837+0000 7f923e7af640 1 -- 172.21.15.3:0/598329502 shutdown_connections 2024-03-25T18:29:55.840 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.837+0000 7f923e7af640 1 -- 172.21.15.3:0/598329502 wait complete. 2024-03-25T18:29:55.840 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:29:55.837+0000 7f923e7af640 1 librados: shutdown 2024-03-25T18:29:55.851 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:01.895 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:05:15.623 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:01.898 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:05:15.628 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:01.920 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:05:15.649 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:01.946 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:05:15.638 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:01.955 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:05:15.647 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:01.997 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:05:15.688 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:02.110 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:05:15.839 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:02.117 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:05:15.847 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:02.139 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:05:15.868 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:02.658 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:02.774 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.772+0000 7f7d252cf640 10 monclient: get_monmap_and_config 2024-03-25T18:30:02.774 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.772+0000 7f7d252cf640 10 monclient: build_initial_monmap 2024-03-25T18:30:02.774 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.772+0000 7f7d252cf640 10 monclient: monmap: 2024-03-25T18:30:02.774 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:30:02.774 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:02.774 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:30:02.773623+0000 2024-03-25T18:30:02.774 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:30:02.773623+0000 2024-03-25T18:30:02.774 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:02.775 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:02.775 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:30:02.775 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:30:02.775 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:30:02.775 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:02.775 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.773+0000 7f7d252cf640 1 Processor -- start 2024-03-25T18:30:02.775 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.773+0000 7f7d252cf640 1 -- start start 2024-03-25T18:30:02.775 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.773+0000 7f7d252cf640 10 monclient: init 2024-03-25T18:30:02.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.773+0000 7f7d252cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:02.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.773+0000 7f7d252cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:30:02.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.773+0000 7f7d252cf640 10 monclient(hunting): picked mon.noname-b con 0x7f7d200b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:30:02.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.773+0000 7f7d252cf640 10 monclient(hunting): picked mon.noname-a con 0x7f7d200b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:30:02.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.774+0000 7f7d252cf640 10 monclient(hunting): picked mon.noname-c con 0x7f7d2013c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:30:02.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.774+0000 7f7d252cf640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d200ab0e0 con 0x7f7d200b4670 2024-03-25T18:30:02.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.774+0000 7f7d252cf640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d200ab3c0 con 0x7f7d2013c480 2024-03-25T18:30:02.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.774+0000 7f7d252cf640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d200a9f30 con 0x7f7d200b4ff0 2024-03-25T18:30:02.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.774+0000 7f7d252cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:02.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.774+0000 7f7d252cf640 10 monclient(hunting): authenticate will time out at 7657.034509s 2024-03-25T18:30:02.782 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.781+0000 7f7d1effd640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f7d200b4ff0 0x7f7d2013bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:44974/0 (socket says 172.21.15.3:44974) 2024-03-25T18:30:02.782 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.781+0000 7f7d1effd640 1 -- 172.21.15.3:0/3422442328 learned_addr learned my addr 172.21.15.3:0/3422442328 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:30:02.794 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.792+0000 7f7d1dffb640 1 -- 172.21.15.3:0/3422442328 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 671190620 0 0) 0x7f7d200a9f30 con 0x7f7d200b4ff0 2024-03-25T18:30:02.794 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.792+0000 7f7d1dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:02.794 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.792+0000 7f7d1dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:02.794 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.792+0000 7f7d1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:02.794 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.792+0000 7f7d1dffb640 10 monclient(hunting): my global_id is 4424 2024-03-25T18:30:02.794 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.792+0000 7f7d1dffb640 1 -- 172.21.15.3:0/3422442328 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d08003520 con 0x7f7d200b4ff0 2024-03-25T18:30:02.796 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.795+0000 7f7d1dffb640 1 -- 172.21.15.3:0/3422442328 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3190942851 0 0) 0x7f7d200ab3c0 con 0x7f7d2013c480 2024-03-25T18:30:02.796 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.795+0000 7f7d1dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:02.796 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.795+0000 7f7d1dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:02.796 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.795+0000 7f7d1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:02.796 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.795+0000 7f7d1dffb640 10 monclient(hunting): my global_id is 4414 2024-03-25T18:30:02.796 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.795+0000 7f7d1dffb640 1 -- 172.21.15.3:0/3422442328 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d200a9f30 con 0x7f7d2013c480 2024-03-25T18:30:02.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.798+0000 7f7d1dffb640 1 -- 172.21.15.3:0/3422442328 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1502512939 0 0) 0x7f7d200ab0e0 con 0x7f7d200b4670 2024-03-25T18:30:02.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.798+0000 7f7d1dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:02.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.798+0000 7f7d1dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:02.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.798+0000 7f7d1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:02.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.798+0000 7f7d1dffb640 10 monclient(hunting): my global_id is 4455 2024-03-25T18:30:02.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.798+0000 7f7d1dffb640 1 -- 172.21.15.3:0/3422442328 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d200ab3c0 con 0x7f7d200b4670 2024-03-25T18:30:02.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.803+0000 7f7d1dffb640 1 -- 172.21.15.3:0/3422442328 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2979157 0 0) 0x7f7d08003520 con 0x7f7d200b4ff0 2024-03-25T18:30:02.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.803+0000 7f7d1dffb640 1 -- 172.21.15.3:0/3422442328 >> v1:172.21.15.16:6789/0 conn(0x7f7d2013c480 legacy=0x7f7d2014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:02.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.803+0000 7f7d1dffb640 1 -- 172.21.15.3:0/3422442328 >> v1:172.21.15.3:6789/0 conn(0x7f7d200b4670 legacy=0x7f7d201425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:02.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.803+0000 7f7d1dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:30:02.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.803+0000 7f7d1dffb640 1 monclient: found mon.noname-b 2024-03-25T18:30:02.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.803+0000 7f7d1dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:02.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.803+0000 7f7d1dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-25T18:30:02.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.803+0000 7f7d1dffb640 1 -- 172.21.15.3:0/3422442328 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d200a9b70 con 0x7f7d200b4ff0 2024-03-25T18:30:02.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.803+0000 7f7d1dffb640 10 monclient: _finish_auth 0 2024-03-25T18:30:02.806 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.803+0000 7f7d1dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:30:02.806 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.803+0000 7f7d1dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:02.806 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.803+0000 7f7d1dffb640 1 -- 172.21.15.3:0/3422442328 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7d10001690 con 0x7f7d200b4ff0 2024-03-25T18:30:02.806 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.803+0000 7f7d1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:02.806 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.803+0000 7f7d1dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:02.806 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.803+0000 7f7d1dffb640 10 monclient: dump: 2024-03-25T18:30:02.806 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:02.806 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:02.806 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:02.806 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:02.806 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:02.806 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:02.807 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:02.807 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:02.807 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:02.807 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:02.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.803+0000 7f7d252cf640 5 monclient: authenticate success, global_id 4424 2024-03-25T18:30:02.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.803+0000 7f7d252cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:02.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.833+0000 7f7d1dffb640 1 -- 172.21.15.3:0/3422442328 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d100017f0 con 0x7f7d200b4ff0 2024-03-25T18:30:02.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.833+0000 7f7d1dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:02.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.833+0000 7f7d1dffb640 1 -- 172.21.15.3:0/3422442328 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7d10001c00 con 0x7f7d200b4ff0 2024-03-25T18:30:02.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.834+0000 7f7d252cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:02.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.834+0000 7f7d252cf640 4 set_mon_vals no callback set 2024-03-25T18:30:02.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.834+0000 7f7d1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:02.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.834+0000 7f7d1dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:02.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.834+0000 7f7d1dffb640 10 monclient: dump: 2024-03-25T18:30:02.835 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:02.835 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:02.835 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:02.835 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:02.836 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:02.836 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:02.836 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:02.836 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:02.836 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:02.836 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:02.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.834+0000 7f7d252cf640 10 monclient: shutdown 2024-03-25T18:30:02.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.834+0000 7f7d252cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:02.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.834+0000 7f7d252cf640 1 -- 172.21.15.3:0/3422442328 >> v1:172.21.15.3:6790/0 conn(0x7f7d200b4ff0 legacy=0x7f7d2013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:02.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.834+0000 7f7d252cf640 1 -- 172.21.15.3:0/3422442328 shutdown_connections 2024-03-25T18:30:02.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.834+0000 7f7d252cf640 1 -- 172.21.15.3:0/3422442328 >> 172.21.15.3:0/3422442328 conn(0x7f7d200617a0 msgr2=0x7f7d20061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:02.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.834+0000 7f7d252cf640 1 -- 172.21.15.3:0/3422442328 shutdown_connections 2024-03-25T18:30:02.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.835+0000 7f7d252cf640 1 -- 172.21.15.3:0/3422442328 wait complete. 2024-03-25T18:30:02.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.835+0000 7f7d252cf640 10 monclient: build_initial_monmap 2024-03-25T18:30:02.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.835+0000 7f7d252cf640 10 monclient: monmap: 2024-03-25T18:30:02.838 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:30:02.838 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:02.838 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:30:02.838 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:30:02.838 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:02.838 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:30:02.838 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:30:02.838 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:30:02.838 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:30:02.838 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:02.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.835+0000 7f7d252cf640 1 librados: starting msgr at 2024-03-25T18:30:02.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.835+0000 7f7d252cf640 1 librados: starting objecter 2024-03-25T18:30:02.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.836+0000 7f7d252cf640 1 Processor -- start 2024-03-25T18:30:02.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.836+0000 7f7d252cf640 1 -- start start 2024-03-25T18:30:02.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.836+0000 7f7d252cf640 1 librados: setting wanted keys 2024-03-25T18:30:02.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.836+0000 7f7d252cf640 1 librados: calling monclient init 2024-03-25T18:30:02.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.836+0000 7f7d252cf640 10 monclient: init 2024-03-25T18:30:02.840 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.836+0000 7f7d252cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:02.840 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.836+0000 7f7d252cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:30:02.840 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.836+0000 7f7d252cf640 10 monclient(hunting): picked mon.noname-c con 0x7f7d200b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:30:02.840 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.837+0000 7f7d252cf640 10 monclient(hunting): picked mon.noname-b con 0x7f7d200b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:30:02.840 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.837+0000 7f7d252cf640 10 monclient(hunting): picked mon.noname-a con 0x7f7d2013c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:30:02.840 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.837+0000 7f7d252cf640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d201464e0 con 0x7f7d2013c480 2024-03-25T18:30:02.840 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.837+0000 7f7d252cf640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d20143940 con 0x7f7d200b4ff0 2024-03-25T18:30:02.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.837+0000 7f7d252cf640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d20144c50 con 0x7f7d200b4670 2024-03-25T18:30:02.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.837+0000 7f7d252cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:02.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.837+0000 7f7d252cf640 10 monclient(hunting): authenticate will time out at 7657.097496s 2024-03-25T18:30:02.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.846+0000 7f7d1f7fe640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f7d2013c480 0x7f7d201e2170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:46714/0 (socket says 172.21.15.3:46714) 2024-03-25T18:30:02.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.846+0000 7f7d1f7fe640 1 -- 172.21.15.3:0/2928920902 learned_addr learned my addr 172.21.15.3:0/2928920902 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:30:02.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.847+0000 7f7d077fe640 10 client.?.objecter ms_handle_connect 0x7f7d2013c480 2024-03-25T18:30:02.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.847+0000 7f7d077fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:02.852 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.851+0000 7f7d077fe640 10 client.?.objecter ms_handle_connect 0x7f7d200b4ff0 2024-03-25T18:30:02.852 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.851+0000 7f7d077fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:02.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.851+0000 7f7d077fe640 10 client.?.objecter ms_handle_connect 0x7f7d200b4670 2024-03-25T18:30:02.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.852+0000 7f7d077fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:02.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.855+0000 7f7d077fe640 1 -- 172.21.15.3:0/2928920902 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4042225786 0 0) 0x7f7d201464e0 con 0x7f7d2013c480 2024-03-25T18:30:02.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.855+0000 7f7d077fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:02.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.855+0000 7f7d077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:02.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.855+0000 7f7d077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:02.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.855+0000 7f7d077fe640 10 monclient(hunting): my global_id is 4458 2024-03-25T18:30:02.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.855+0000 7f7d077fe640 1 -- 172.21.15.3:0/2928920902 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d0c004eb0 con 0x7f7d2013c480 2024-03-25T18:30:02.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.860+0000 7f7d077fe640 1 -- 172.21.15.3:0/2928920902 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 548435478 0 0) 0x7f7d20143940 con 0x7f7d200b4ff0 2024-03-25T18:30:02.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.860+0000 7f7d077fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:02.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.860+0000 7f7d077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:02.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.860+0000 7f7d077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:02.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.860+0000 7f7d077fe640 10 monclient(hunting): my global_id is 4417 2024-03-25T18:30:02.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.860+0000 7f7d077fe640 1 -- 172.21.15.3:0/2928920902 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d201464e0 con 0x7f7d200b4ff0 2024-03-25T18:30:02.862 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.861+0000 7f7d077fe640 1 -- 172.21.15.3:0/2928920902 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3606328258 0 0) 0x7f7d20144c50 con 0x7f7d200b4670 2024-03-25T18:30:02.863 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.861+0000 7f7d077fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:02.863 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.861+0000 7f7d077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:02.863 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.861+0000 7f7d077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:02.863 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.861+0000 7f7d077fe640 10 monclient(hunting): my global_id is 4427 2024-03-25T18:30:02.863 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.861+0000 7f7d077fe640 1 -- 172.21.15.3:0/2928920902 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d20143940 con 0x7f7d200b4670 2024-03-25T18:30:02.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.870+0000 7f7d077fe640 1 -- 172.21.15.3:0/2928920902 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2086496561 0 0) 0x7f7d0c004eb0 con 0x7f7d2013c480 2024-03-25T18:30:02.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.870+0000 7f7d077fe640 1 -- 172.21.15.3:0/2928920902 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d20144c50 con 0x7f7d2013c480 2024-03-25T18:30:02.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.870+0000 7f7d077fe640 1 -- 172.21.15.3:0/2928920902 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7d18001c10 con 0x7f7d2013c480 2024-03-25T18:30:02.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.870+0000 7f7d077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:02.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.870+0000 7f7d077fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:02.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.870+0000 7f7d077fe640 10 monclient(hunting): dump: 2024-03-25T18:30:02.872 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:02.872 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:02.872 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:02.872 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:02.872 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:02.872 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:02.873 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:02.873 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:02.873 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:02.873 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:02.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.870+0000 7f7d077fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:02.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.876+0000 7f7d077fe640 1 -- 172.21.15.3:0/2928920902 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1748636886 0 0) 0x7f7d20143940 con 0x7f7d200b4670 2024-03-25T18:30:02.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.876+0000 7f7d077fe640 1 -- 172.21.15.3:0/2928920902 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d0c004eb0 con 0x7f7d200b4670 2024-03-25T18:30:02.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.876+0000 7f7d077fe640 1 -- 172.21.15.3:0/2928920902 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7d10001b00 con 0x7f7d200b4670 2024-03-25T18:30:02.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.876+0000 7f7d077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:02.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.876+0000 7f7d077fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:02.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.876+0000 7f7d077fe640 10 monclient(hunting): dump: 2024-03-25T18:30:02.878 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:02.878 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:02.878 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:02.878 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:02.878 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:02.878 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:02.878 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:02.878 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:02.879 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:02.879 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:02.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.883+0000 7f7d077fe640 1 -- 172.21.15.3:0/2928920902 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3283487261 0 0) 0x7f7d0c004eb0 con 0x7f7d200b4670 2024-03-25T18:30:02.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.883+0000 7f7d077fe640 1 -- 172.21.15.3:0/2928920902 >> v1:172.21.15.16:6789/0 conn(0x7f7d200b4ff0 legacy=0x7f7d20145dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:02.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.883+0000 7f7d077fe640 1 -- 172.21.15.3:0/2928920902 >> v1:172.21.15.3:6789/0 conn(0x7f7d2013c480 legacy=0x7f7d201e2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:02.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.883+0000 7f7d077fe640 10 monclient: _finish_hunting 0 2024-03-25T18:30:02.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.883+0000 7f7d077fe640 1 monclient: found mon.c 2024-03-25T18:30:02.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.883+0000 7f7d077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:02.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.883+0000 7f7d077fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:30:02.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.883+0000 7f7d077fe640 1 -- 172.21.15.3:0/2928920902 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d20144f30 con 0x7f7d200b4670 2024-03-25T18:30:02.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.883+0000 7f7d077fe640 10 monclient: _finish_auth 0 2024-03-25T18:30:02.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.883+0000 7f7d077fe640 10 monclient: _check_auth_tickets 2024-03-25T18:30:02.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.883+0000 7f7d077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:02.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.883+0000 7f7d252cf640 5 monclient: authenticate success, global_id 4427 2024-03-25T18:30:02.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.883+0000 7f7d252cf640 10 monclient: _renew_subs 2024-03-25T18:30:02.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.883+0000 7f7d252cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:30:02.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.883+0000 7f7d252cf640 1 -- 172.21.15.3:0/2928920902 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d2013f820 con 0x7f7d200b4670 2024-03-25T18:30:02.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.883+0000 7f7d252cf640 10 client.4427.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:02.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.883+0000 7f7d252cf640 10 monclient: _renew_subs 2024-03-25T18:30:02.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.883+0000 7f7d252cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:30:02.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.883+0000 7f7d252cf640 1 -- 172.21.15.3:0/2928920902 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d20143660 con 0x7f7d200b4670 2024-03-25T18:30:02.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.883+0000 7f7d252cf640 1 librados: init done 2024-03-25T18:30:02.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.884+0000 7f7d252cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:02.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.884+0000 7f7d252cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:02.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.884+0000 7f7d252cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:30:02.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.884+0000 7f7d252cf640 1 -- 172.21.15.3:0/2928920902 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d2006dbd0 con 0x7f7d200b4670 2024-03-25T18:30:02.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.890+0000 7f7d077fe640 1 -- 172.21.15.3:0/2928920902 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d10001c80 con 0x7f7d200b4670 2024-03-25T18:30:02.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.890+0000 7f7d077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:02.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.890+0000 7f7d1d7fa640 4 set_mon_vals no callback set 2024-03-25T18:30:02.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.931+0000 7f7d077fe640 1 -- 172.21.15.3:0/2928920902 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7d10004c30 con 0x7f7d200b4670 2024-03-25T18:30:02.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.931+0000 7f7d077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:02.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.931+0000 7f7d077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:02.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:02.931+0000 7f7d077fe640 10 monclient: dump: 2024-03-25T18:30:02.932 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:02.932 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:02.932 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:02.933 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:02.933 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:02.933 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:02.933 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:02.933 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:02.933 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:02.933 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:03.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:03.201+0000 7f7d077fe640 1 -- 172.21.15.3:0/2928920902 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f7d1001ac00 con 0x7f7d200b4670 2024-03-25T18:30:03.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:03.202+0000 7f7d1e7fc640 10 monclient: get_auth_request con 0x7f7d0c06df70 auth_method 0 2024-03-25T18:30:03.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:03.202+0000 7f7d077fe640 10 client.4427.objecter ms_handle_connect 0x7f7d0c06df70 2024-03-25T18:30:03.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:03.212+0000 7f7d077fe640 1 -- 172.21.15.3:0/2928920902 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f7d100547e0 con 0x7f7d200b4670 2024-03-25T18:30:03.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:03.212+0000 7f7d077fe640 10 client.4427.objecter ms_dispatch 0x7f7d20000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:03.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:03.212+0000 7f7d077fe640 3 client.4427.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:03.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:03.212+0000 7f7d077fe640 3 client.4427.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:03.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:03.212+0000 7f7d077fe640 20 client.4427.objecter dump_active .. 0 homeless 2024-03-25T18:30:03.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:03.212+0000 7f7d077fe640 1 -- 172.21.15.3:0/2928920902 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7d10050120 con 0x7f7d200b4670 2024-03-25T18:30:03.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:03.212+0000 7f7d077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:03.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:03.212+0000 7f7d077fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:03.473 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:03.472+0000 7f7d252cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:03.473 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:03.472+0000 7f7d252cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:03.473 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:03.472+0000 7f7d252cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:30:03.473 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:03.472+0000 7f7d252cf640 1 -- 172.21.15.3:0/2928920902 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d200688c0 con 0x7f7d200b4670 2024-03-25T18:30:03.494 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:03.493+0000 7f7d077fe640 1 -- 172.21.15.3:0/2928920902 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f7d10052220 con 0x7f7d200b4670 2024-03-25T18:30:03.494 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:03.493+0000 7f7d077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:03.494 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:03.493+0000 7f7d077fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:03.494 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:30:03.495 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:03.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:03.496+0000 7f7d252cf640 10 librados: watch_flush enter 2024-03-25T18:30:03.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:03.496+0000 7f7d252cf640 10 librados: watch_flush exit 2024-03-25T18:30:03.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:03.496+0000 7f7d252cf640 20 client.4427.objecter shutdown clearing up homeless session... 2024-03-25T18:30:03.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:03.496+0000 7f7d252cf640 10 client.4427.objecter successfully canceled tick 2024-03-25T18:30:03.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:03.496+0000 7f7d252cf640 1 -- 172.21.15.3:0/2928920902 >> v1:172.21.15.3:6800/4195140966 conn(0x7f7d0c06df70 legacy=0x7f7d0c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:03.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:03.496+0000 7f7d252cf640 10 monclient: shutdown 2024-03-25T18:30:03.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:03.496+0000 7f7d252cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:03.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:03.496+0000 7f7d252cf640 1 -- 172.21.15.3:0/2928920902 >> v1:172.21.15.3:6790/0 conn(0x7f7d200b4670 legacy=0x7f7d201456c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:03.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:03.496+0000 7f7d252cf640 1 -- 172.21.15.3:0/2928920902 shutdown_connections 2024-03-25T18:30:03.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:03.496+0000 7f7d252cf640 1 -- 172.21.15.3:0/2928920902 >> 172.21.15.3:0/2928920902 conn(0x7f7d200617a0 msgr2=0x7f7d201407f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:03.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:03.496+0000 7f7d252cf640 1 -- 172.21.15.3:0/2928920902 shutdown_connections 2024-03-25T18:30:03.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:03.496+0000 7f7d252cf640 1 -- 172.21.15.3:0/2928920902 wait complete. 2024-03-25T18:30:03.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:03.496+0000 7f7d252cf640 1 librados: shutdown 2024-03-25T18:30:03.511 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:10.318 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:10.436 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.435+0000 7fb7e342a640 10 monclient: get_monmap_and_config 2024-03-25T18:30:10.436 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.435+0000 7fb7e342a640 10 monclient: build_initial_monmap 2024-03-25T18:30:10.436 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.435+0000 7fb7e342a640 10 monclient: monmap: 2024-03-25T18:30:10.437 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:30:10.437 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:10.437 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:30:10.436376+0000 2024-03-25T18:30:10.437 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:30:10.436376+0000 2024-03-25T18:30:10.437 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:10.437 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:10.437 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:30:10.437 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:30:10.437 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:30:10.437 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:10.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.435+0000 7fb7e342a640 1 Processor -- start 2024-03-25T18:30:10.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.436+0000 7fb7e342a640 1 -- start start 2024-03-25T18:30:10.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.436+0000 7fb7e342a640 10 monclient: init 2024-03-25T18:30:10.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.436+0000 7fb7e342a640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:10.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.436+0000 7fb7e342a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:30:10.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.437+0000 7fb7e342a640 10 monclient(hunting): picked mon.noname-b con 0x7fb7dc0b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:30:10.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.437+0000 7fb7e342a640 10 monclient(hunting): picked mon.noname-c con 0x7fb7dc0b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:30:10.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.437+0000 7fb7e342a640 10 monclient(hunting): picked mon.noname-a con 0x7fb7dc13c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:30:10.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.437+0000 7fb7e342a640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7dc0ab0e0 con 0x7fb7dc13c480 2024-03-25T18:30:10.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.437+0000 7fb7e342a640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7dc0ab3c0 con 0x7fb7dc0b4670 2024-03-25T18:30:10.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.437+0000 7fb7e342a640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7dc0a9f30 con 0x7fb7dc0b4ff0 2024-03-25T18:30:10.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.437+0000 7fb7e342a640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:10.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.437+0000 7fb7e342a640 10 monclient(hunting): authenticate will time out at 7664.697612s 2024-03-25T18:30:10.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.444+0000 7fb7e19c7640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fb7dc13c480 0x7fb7dc14dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56090/0 (socket says 172.21.15.3:56090) 2024-03-25T18:30:10.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.444+0000 7fb7e19c7640 1 -- 172.21.15.3:0/3735381141 learned_addr learned my addr 172.21.15.3:0/3735381141 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:30:10.456 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.455+0000 7fb7d3fff640 1 -- 172.21.15.3:0/3735381141 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1964294020 0 0) 0x7fb7dc0ab0e0 con 0x7fb7dc13c480 2024-03-25T18:30:10.456 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.455+0000 7fb7d3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:10.456 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.455+0000 7fb7d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:10.456 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.455+0000 7fb7d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:10.456 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.455+0000 7fb7d3fff640 10 monclient(hunting): my global_id is 4461 2024-03-25T18:30:10.457 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.455+0000 7fb7d3fff640 1 -- 172.21.15.3:0/3735381141 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7c8003520 con 0x7fb7dc13c480 2024-03-25T18:30:10.463 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.462+0000 7fb7d3fff640 1 -- 172.21.15.3:0/3735381141 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2142297094 0 0) 0x7fb7dc0a9f30 con 0x7fb7dc0b4ff0 2024-03-25T18:30:10.463 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.462+0000 7fb7d3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:10.464 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.462+0000 7fb7d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:10.464 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.462+0000 7fb7d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:10.464 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.462+0000 7fb7d3fff640 10 monclient(hunting): my global_id is 4430 2024-03-25T18:30:10.464 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.462+0000 7fb7d3fff640 1 -- 172.21.15.3:0/3735381141 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7dc0ab0e0 con 0x7fb7dc0b4ff0 2024-03-25T18:30:10.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.464+0000 7fb7d3fff640 1 -- 172.21.15.3:0/3735381141 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2073698394 0 0) 0x7fb7dc0ab3c0 con 0x7fb7dc0b4670 2024-03-25T18:30:10.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.464+0000 7fb7d3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:10.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.464+0000 7fb7d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:10.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.464+0000 7fb7d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:10.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.464+0000 7fb7d3fff640 10 monclient(hunting): my global_id is 4420 2024-03-25T18:30:10.466 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.464+0000 7fb7d3fff640 1 -- 172.21.15.3:0/3735381141 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7dc0a9f30 con 0x7fb7dc0b4670 2024-03-25T18:30:10.474 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.473+0000 7fb7d3fff640 1 -- 172.21.15.3:0/3735381141 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1578300562 0 0) 0x7fb7dc0a9f30 con 0x7fb7dc0b4670 2024-03-25T18:30:10.474 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.473+0000 7fb7d3fff640 1 -- 172.21.15.3:0/3735381141 >> v1:172.21.15.3:6790/0 conn(0x7fb7dc0b4ff0 legacy=0x7fb7dc13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:10.474 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.473+0000 7fb7d3fff640 1 -- 172.21.15.3:0/3735381141 >> v1:172.21.15.3:6789/0 conn(0x7fb7dc13c480 legacy=0x7fb7dc14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:10.475 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.473+0000 7fb7d3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:30:10.475 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.473+0000 7fb7d3fff640 1 monclient: found mon.noname-c 2024-03-25T18:30:10.475 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.473+0000 7fb7d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:10.475 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.473+0000 7fb7d3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:30:10.475 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.473+0000 7fb7d3fff640 1 -- 172.21.15.3:0/3735381141 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7dc0a9b70 con 0x7fb7dc0b4670 2024-03-25T18:30:10.475 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.473+0000 7fb7d3fff640 10 monclient: _finish_auth 0 2024-03-25T18:30:10.475 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.473+0000 7fb7d3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:30:10.475 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.473+0000 7fb7d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:10.475 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.473+0000 7fb7d3fff640 1 -- 172.21.15.3:0/3735381141 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fb7cc0015b0 con 0x7fb7dc0b4670 2024-03-25T18:30:10.475 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.473+0000 7fb7d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:10.475 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.473+0000 7fb7d3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:10.475 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.473+0000 7fb7d3fff640 10 monclient: dump: 2024-03-25T18:30:10.475 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:10.476 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:10.476 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:10.476 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:10.476 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:10.476 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:10.476 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:10.476 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:10.476 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:10.476 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:10.476 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.473+0000 7fb7e342a640 5 monclient: authenticate success, global_id 4420 2024-03-25T18:30:10.476 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.473+0000 7fb7e342a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:10.494 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.493+0000 7fb7d3fff640 1 -- 172.21.15.3:0/3735381141 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7cc001710 con 0x7fb7dc0b4670 2024-03-25T18:30:10.494 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.493+0000 7fb7d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:10.494 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.493+0000 7fb7d3fff640 1 -- 172.21.15.3:0/3735381141 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fb7cc001b60 con 0x7fb7dc0b4670 2024-03-25T18:30:10.494 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.493+0000 7fb7e342a640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:10.494 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.493+0000 7fb7e342a640 4 set_mon_vals no callback set 2024-03-25T18:30:10.494 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.493+0000 7fb7d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:10.494 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.493+0000 7fb7d3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:10.494 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.493+0000 7fb7d3fff640 10 monclient: dump: 2024-03-25T18:30:10.495 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:10.495 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:10.495 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:10.495 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:10.495 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:10.495 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:10.495 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:10.495 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:10.495 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:10.495 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:10.496 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.494+0000 7fb7e342a640 10 monclient: shutdown 2024-03-25T18:30:10.496 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.494+0000 7fb7e342a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:10.497 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.494+0000 7fb7e342a640 1 -- 172.21.15.3:0/3735381141 >> v1:172.21.15.16:6789/0 conn(0x7fb7dc0b4670 legacy=0x7fb7dc1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:10.497 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.494+0000 7fb7e342a640 1 -- 172.21.15.3:0/3735381141 shutdown_connections 2024-03-25T18:30:10.497 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.494+0000 7fb7e342a640 1 -- 172.21.15.3:0/3735381141 >> 172.21.15.3:0/3735381141 conn(0x7fb7dc0617a0 msgr2=0x7fb7dc061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:10.497 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.494+0000 7fb7e342a640 1 -- 172.21.15.3:0/3735381141 shutdown_connections 2024-03-25T18:30:10.497 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.494+0000 7fb7e342a640 1 -- 172.21.15.3:0/3735381141 wait complete. 2024-03-25T18:30:10.497 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.494+0000 7fb7e342a640 10 monclient: build_initial_monmap 2024-03-25T18:30:10.497 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.494+0000 7fb7e342a640 10 monclient: monmap: 2024-03-25T18:30:10.497 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:30:10.497 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:10.497 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:30:10.497 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:30:10.497 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:10.497 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:30:10.497 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:30:10.497 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:30:10.498 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:30:10.498 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:10.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.495+0000 7fb7e342a640 1 librados: starting msgr at 2024-03-25T18:30:10.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.495+0000 7fb7e342a640 1 librados: starting objecter 2024-03-25T18:30:10.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.495+0000 7fb7e342a640 1 Processor -- start 2024-03-25T18:30:10.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.495+0000 7fb7e342a640 1 -- start start 2024-03-25T18:30:10.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.495+0000 7fb7e342a640 1 librados: setting wanted keys 2024-03-25T18:30:10.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.495+0000 7fb7e342a640 1 librados: calling monclient init 2024-03-25T18:30:10.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.495+0000 7fb7e342a640 10 monclient: init 2024-03-25T18:30:10.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.496+0000 7fb7e342a640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:10.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.496+0000 7fb7e342a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:30:10.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.496+0000 7fb7e342a640 10 monclient(hunting): picked mon.noname-c con 0x7fb7dc0b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:30:10.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.496+0000 7fb7e342a640 10 monclient(hunting): picked mon.noname-a con 0x7fb7dc0b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:30:10.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.496+0000 7fb7e342a640 10 monclient(hunting): picked mon.noname-b con 0x7fb7dc13c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:30:10.500 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.496+0000 7fb7e342a640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7dc145a00 con 0x7fb7dc0b4ff0 2024-03-25T18:30:10.500 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.496+0000 7fb7e342a640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7dc1456d0 con 0x7fb7dc13c480 2024-03-25T18:30:10.500 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.496+0000 7fb7e342a640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7dc144fd0 con 0x7fb7dc0b4670 2024-03-25T18:30:10.500 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.496+0000 7fb7e342a640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:10.500 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.496+0000 7fb7e342a640 10 monclient(hunting): authenticate will time out at 7664.756770s 2024-03-25T18:30:10.510 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.509+0000 7fb7e19c7640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb7dc13c480 0x7fb7dc1e38d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:48078/0 (socket says 172.21.15.3:48078) 2024-03-25T18:30:10.510 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.509+0000 7fb7e19c7640 1 -- 172.21.15.3:0/4269372153 learned_addr learned my addr 172.21.15.3:0/4269372153 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:30:10.512 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.511+0000 7fb7d1ffb640 10 client.?.objecter ms_handle_connect 0x7fb7dc0b4670 2024-03-25T18:30:10.513 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.511+0000 7fb7d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:10.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.514+0000 7fb7d1ffb640 10 client.?.objecter ms_handle_connect 0x7fb7dc13c480 2024-03-25T18:30:10.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.514+0000 7fb7d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:10.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.517+0000 7fb7d1ffb640 10 client.?.objecter ms_handle_connect 0x7fb7dc0b4ff0 2024-03-25T18:30:10.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.517+0000 7fb7d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:10.530 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.529+0000 7fb7d1ffb640 1 -- 172.21.15.3:0/4269372153 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3397461243 0 0) 0x7fb7dc144fd0 con 0x7fb7dc0b4670 2024-03-25T18:30:10.530 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.529+0000 7fb7d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:10.530 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.529+0000 7fb7d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:10.530 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.529+0000 7fb7d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:10.530 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.529+0000 7fb7d1ffb640 10 monclient(hunting): my global_id is 4433 2024-03-25T18:30:10.530 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.529+0000 7fb7d1ffb640 1 -- 172.21.15.3:0/4269372153 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7c0004eb0 con 0x7fb7dc0b4670 2024-03-25T18:30:10.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.531+0000 7fb7d1ffb640 1 -- 172.21.15.3:0/4269372153 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 510729244 0 0) 0x7fb7dc145a00 con 0x7fb7dc0b4ff0 2024-03-25T18:30:10.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.531+0000 7fb7d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:10.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.531+0000 7fb7d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:10.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.531+0000 7fb7d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:10.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.531+0000 7fb7d1ffb640 10 monclient(hunting): my global_id is 4464 2024-03-25T18:30:10.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.531+0000 7fb7d1ffb640 1 -- 172.21.15.3:0/4269372153 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7dc144fd0 con 0x7fb7dc0b4ff0 2024-03-25T18:30:10.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.533+0000 7fb7d1ffb640 1 -- 172.21.15.3:0/4269372153 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2600130628 0 0) 0x7fb7dc1456d0 con 0x7fb7dc13c480 2024-03-25T18:30:10.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.533+0000 7fb7d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:10.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.533+0000 7fb7d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:10.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.533+0000 7fb7d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:10.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.533+0000 7fb7d1ffb640 10 monclient(hunting): my global_id is 4423 2024-03-25T18:30:10.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.533+0000 7fb7d1ffb640 1 -- 172.21.15.3:0/4269372153 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7dc145a00 con 0x7fb7dc13c480 2024-03-25T18:30:10.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.551+0000 7fb7d1ffb640 1 -- 172.21.15.3:0/4269372153 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4188354103 0 0) 0x7fb7c0004eb0 con 0x7fb7dc0b4670 2024-03-25T18:30:10.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.551+0000 7fb7d1ffb640 1 -- 172.21.15.3:0/4269372153 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7dc1456d0 con 0x7fb7dc0b4670 2024-03-25T18:30:10.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.551+0000 7fb7d1ffb640 1 -- 172.21.15.3:0/4269372153 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fb7d8001930 con 0x7fb7dc0b4670 2024-03-25T18:30:10.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.551+0000 7fb7d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:10.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.551+0000 7fb7d1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:10.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.551+0000 7fb7d1ffb640 10 monclient(hunting): dump: 2024-03-25T18:30:10.554 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:10.554 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:10.554 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:10.554 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:10.554 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:10.554 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:10.554 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:10.554 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:10.554 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:10.554 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:10.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.551+0000 7fb7d1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:10.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.555+0000 7fb7d1ffb640 1 -- 172.21.15.3:0/4269372153 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2748294127 0 0) 0x7fb7dc145a00 con 0x7fb7dc13c480 2024-03-25T18:30:10.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.556+0000 7fb7d1ffb640 1 -- 172.21.15.3:0/4269372153 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7c0004eb0 con 0x7fb7dc13c480 2024-03-25T18:30:10.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.556+0000 7fb7d1ffb640 1 -- 172.21.15.3:0/4269372153 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fb7d4001ab0 con 0x7fb7dc13c480 2024-03-25T18:30:10.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.556+0000 7fb7d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:10.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.556+0000 7fb7d1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:10.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.556+0000 7fb7d1ffb640 10 monclient(hunting): dump: 2024-03-25T18:30:10.557 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:10.557 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:10.557 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:10.557 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:10.557 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:10.557 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:10.558 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:10.558 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:10.558 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:10.558 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:10.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.560+0000 7fb7d1ffb640 1 -- 172.21.15.3:0/4269372153 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 545924345 0 0) 0x7fb7c0004eb0 con 0x7fb7dc13c480 2024-03-25T18:30:10.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.560+0000 7fb7d1ffb640 1 -- 172.21.15.3:0/4269372153 >> v1:172.21.15.3:6790/0 conn(0x7fb7dc0b4670 legacy=0x7fb7dc145e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:10.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.560+0000 7fb7d1ffb640 1 -- 172.21.15.3:0/4269372153 >> v1:172.21.15.3:6789/0 conn(0x7fb7dc0b4ff0 legacy=0x7fb7dc130200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:10.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.560+0000 7fb7d1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:30:10.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.560+0000 7fb7d1ffb640 1 monclient: found mon.b 2024-03-25T18:30:10.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.560+0000 7fb7d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:10.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.560+0000 7fb7d1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:30:10.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.560+0000 7fb7d1ffb640 1 -- 172.21.15.3:0/4269372153 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7dc1452b0 con 0x7fb7dc13c480 2024-03-25T18:30:10.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.560+0000 7fb7d1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:30:10.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.560+0000 7fb7d1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:30:10.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.560+0000 7fb7d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:10.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.560+0000 7fb7e342a640 5 monclient: authenticate success, global_id 4423 2024-03-25T18:30:10.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.560+0000 7fb7e342a640 10 monclient: _renew_subs 2024-03-25T18:30:10.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.560+0000 7fb7e342a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:30:10.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.560+0000 7fb7e342a640 1 -- 172.21.15.3:0/4269372153 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7dc13cb20 con 0x7fb7dc13c480 2024-03-25T18:30:10.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.560+0000 7fb7e342a640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:10.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.560+0000 7fb7e342a640 10 monclient: _renew_subs 2024-03-25T18:30:10.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.560+0000 7fb7e342a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:30:10.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.560+0000 7fb7e342a640 1 -- 172.21.15.3:0/4269372153 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7dc146780 con 0x7fb7dc13c480 2024-03-25T18:30:10.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.560+0000 7fb7e342a640 1 librados: init done 2024-03-25T18:30:10.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.562+0000 7fb7e342a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:10.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.562+0000 7fb7e342a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:10.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.562+0000 7fb7e342a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:30:10.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.562+0000 7fb7e342a640 1 -- 172.21.15.3:0/4269372153 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7dc0b3c50 con 0x7fb7dc13c480 2024-03-25T18:30:10.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.865+0000 7fb7d1ffb640 1 -- 172.21.15.3:0/4269372153 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7d4001fc0 con 0x7fb7dc13c480 2024-03-25T18:30:10.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.865+0000 7fb7d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:10.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.865+0000 7fb7d1ffb640 1 -- 172.21.15.3:0/4269372153 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fb7d4004d00 con 0x7fb7dc13c480 2024-03-25T18:30:10.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.865+0000 7fb7d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:10.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.865+0000 7fb7d1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:10.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.865+0000 7fb7d1ffb640 10 monclient: dump: 2024-03-25T18:30:10.867 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:10.867 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:10.868 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:10.868 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:10.868 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:10.868 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:10.868 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:10.868 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:10.868 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:10.868 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:10.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.865+0000 7fb7d37fe640 4 set_mon_vals no callback set 2024-03-25T18:30:10.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.866+0000 7fb7d1ffb640 1 -- 172.21.15.3:0/4269372153 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7fb7d401acd0 con 0x7fb7dc13c480 2024-03-25T18:30:10.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.867+0000 7fb7e11c6640 10 monclient: get_auth_request con 0x7fb7c006dfb0 auth_method 0 2024-03-25T18:30:10.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.867+0000 7fb7d1ffb640 10 client.4423.objecter ms_handle_connect 0x7fb7c006dfb0 2024-03-25T18:30:10.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.874+0000 7fb7d1ffb640 1 -- 172.21.15.3:0/4269372153 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7fb7d4054a60 con 0x7fb7dc13c480 2024-03-25T18:30:10.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.874+0000 7fb7d1ffb640 10 client.4423.objecter ms_dispatch 0x7fb7dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:10.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.874+0000 7fb7d1ffb640 3 client.4423.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:10.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.874+0000 7fb7d1ffb640 3 client.4423.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:10.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.874+0000 7fb7d1ffb640 20 client.4423.objecter dump_active .. 0 homeless 2024-03-25T18:30:10.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.876+0000 7fb7d1ffb640 1 -- 172.21.15.3:0/4269372153 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb7d40503a0 con 0x7fb7dc13c480 2024-03-25T18:30:10.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.876+0000 7fb7d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:10.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:10.876+0000 7fb7d1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:11.127 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:11.125+0000 7fb7e342a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:11.127 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:11.125+0000 7fb7e342a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:11.127 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:11.125+0000 7fb7e342a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:30:11.127 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:11.125+0000 7fb7e342a640 1 -- 172.21.15.3:0/4269372153 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7dc06dbd0 con 0x7fb7dc13c480 2024-03-25T18:30:11.151 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:11.150+0000 7fb7d1ffb640 1 -- 172.21.15.3:0/4269372153 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7fb7d40524a0 con 0x7fb7dc13c480 2024-03-25T18:30:11.151 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:11.150+0000 7fb7d1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:11.151 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:11.150+0000 7fb7d1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:11.151 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:30:11.151 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:11.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:11.152+0000 7fb7e342a640 10 librados: watch_flush enter 2024-03-25T18:30:11.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:11.152+0000 7fb7e342a640 10 librados: watch_flush exit 2024-03-25T18:30:11.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:11.152+0000 7fb7e342a640 20 client.4423.objecter shutdown clearing up homeless session... 2024-03-25T18:30:11.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:11.152+0000 7fb7e342a640 10 client.4423.objecter successfully canceled tick 2024-03-25T18:30:11.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:11.152+0000 7fb7e342a640 1 -- 172.21.15.3:0/4269372153 >> v1:172.21.15.3:6800/4195140966 conn(0x7fb7c006dfb0 legacy=0x7fb7c0070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:11.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:11.152+0000 7fb7e342a640 10 monclient: shutdown 2024-03-25T18:30:11.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:11.152+0000 7fb7e342a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:11.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:11.152+0000 7fb7e342a640 1 -- 172.21.15.3:0/4269372153 >> v1:172.21.15.16:6789/0 conn(0x7fb7dc13c480 legacy=0x7fb7dc1e38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:11.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:11.152+0000 7fb7e342a640 1 -- 172.21.15.3:0/4269372153 shutdown_connections 2024-03-25T18:30:11.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:11.152+0000 7fb7e342a640 1 -- 172.21.15.3:0/4269372153 >> 172.21.15.3:0/4269372153 conn(0x7fb7dc0617a0 msgr2=0x7fb7dc141ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:11.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:11.152+0000 7fb7e342a640 1 -- 172.21.15.3:0/4269372153 shutdown_connections 2024-03-25T18:30:11.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:11.152+0000 7fb7e342a640 1 -- 172.21.15.3:0/4269372153 wait complete. 2024-03-25T18:30:11.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:11.153+0000 7fb7e342a640 1 librados: shutdown 2024-03-25T18:30:11.166 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:17.973 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:18.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.090+0000 7fc98847b640 10 monclient: get_monmap_and_config 2024-03-25T18:30:18.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.090+0000 7fc98847b640 10 monclient: build_initial_monmap 2024-03-25T18:30:18.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.090+0000 7fc98847b640 10 monclient: monmap: 2024-03-25T18:30:18.092 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:30:18.092 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:18.092 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:30:18.091979+0000 2024-03-25T18:30:18.092 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:30:18.091979+0000 2024-03-25T18:30:18.092 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:18.092 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:18.092 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:30:18.092 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:30:18.092 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:30:18.093 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:18.093 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.091+0000 7fc98847b640 1 Processor -- start 2024-03-25T18:30:18.093 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.091+0000 7fc98847b640 1 -- start start 2024-03-25T18:30:18.093 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.091+0000 7fc98847b640 10 monclient: init 2024-03-25T18:30:18.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.092+0000 7fc98847b640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:18.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.092+0000 7fc98847b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:30:18.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.092+0000 7fc98847b640 10 monclient(hunting): picked mon.noname-c con 0x7fc98006f460 addr v1:172.21.15.16:6789/0 2024-03-25T18:30:18.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.093+0000 7fc98847b640 10 monclient(hunting): picked mon.noname-b con 0x7fc98006f840 addr v1:172.21.15.3:6790/0 2024-03-25T18:30:18.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.093+0000 7fc98847b640 10 monclient(hunting): picked mon.noname-a con 0x7fc9800728f0 addr v1:172.21.15.3:6789/0 2024-03-25T18:30:18.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.093+0000 7fc98847b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc980071a40 con 0x7fc9800728f0 2024-03-25T18:30:18.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.093+0000 7fc98847b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc980071db0 con 0x7fc98006f460 2024-03-25T18:30:18.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.093+0000 7fc98847b640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc980070650 con 0x7fc98006f840 2024-03-25T18:30:18.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.093+0000 7fc98847b640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:18.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.093+0000 7fc98847b640 10 monclient(hunting): authenticate will time out at 7672.353570s 2024-03-25T18:30:18.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.107+0000 7fc986217640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc98006f460 0x7fc980142cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:52816/0 (socket says 172.21.15.3:52816) 2024-03-25T18:30:18.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.107+0000 7fc986217640 1 -- 172.21.15.3:0/4116738540 learned_addr learned my addr 172.21.15.3:0/4116738540 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:30:18.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.124+0000 7fc985215640 1 -- 172.21.15.3:0/4116738540 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3754498085 0 0) 0x7fc980071db0 con 0x7fc98006f460 2024-03-25T18:30:18.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.124+0000 7fc985215640 10 monclient(hunting): _negotiate 2024-03-25T18:30:18.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.124+0000 7fc985215640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:18.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.124+0000 7fc985215640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:18.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.124+0000 7fc985215640 10 monclient(hunting): my global_id is 4426 2024-03-25T18:30:18.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.124+0000 7fc985215640 1 -- 172.21.15.3:0/4116738540 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc96c003520 con 0x7fc98006f460 2024-03-25T18:30:18.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.129+0000 7fc985215640 1 -- 172.21.15.3:0/4116738540 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261375703 0 0) 0x7fc980071a40 con 0x7fc9800728f0 2024-03-25T18:30:18.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.129+0000 7fc985215640 10 monclient(hunting): _negotiate 2024-03-25T18:30:18.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.129+0000 7fc985215640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:18.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.129+0000 7fc985215640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:18.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.129+0000 7fc985215640 10 monclient(hunting): my global_id is 4467 2024-03-25T18:30:18.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.129+0000 7fc985215640 1 -- 172.21.15.3:0/4116738540 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc980071db0 con 0x7fc9800728f0 2024-03-25T18:30:18.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.138+0000 7fc985215640 1 -- 172.21.15.3:0/4116738540 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3050429586 0 0) 0x7fc980070650 con 0x7fc98006f840 2024-03-25T18:30:18.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.138+0000 7fc985215640 10 monclient(hunting): _negotiate 2024-03-25T18:30:18.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.138+0000 7fc985215640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:18.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.138+0000 7fc985215640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:18.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.138+0000 7fc985215640 10 monclient(hunting): my global_id is 4436 2024-03-25T18:30:18.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.138+0000 7fc985215640 1 -- 172.21.15.3:0/4116738540 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc980071a40 con 0x7fc98006f840 2024-03-25T18:30:18.144 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.142+0000 7fc985215640 1 -- 172.21.15.3:0/4116738540 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1561969304 0 0) 0x7fc96c003520 con 0x7fc98006f460 2024-03-25T18:30:18.144 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.142+0000 7fc985215640 1 -- 172.21.15.3:0/4116738540 >> v1:172.21.15.3:6790/0 conn(0x7fc98006f840 legacy=0x7fc9800721e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:18.144 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.142+0000 7fc985215640 1 -- 172.21.15.3:0/4116738540 >> v1:172.21.15.3:6789/0 conn(0x7fc9800728f0 legacy=0x7fc98014a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:18.144 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.142+0000 7fc985215640 10 monclient: _finish_hunting 0 2024-03-25T18:30:18.144 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.142+0000 7fc985215640 1 monclient: found mon.noname-c 2024-03-25T18:30:18.144 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.142+0000 7fc985215640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:18.144 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.142+0000 7fc985215640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:30:18.144 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.142+0000 7fc985215640 1 -- 172.21.15.3:0/4116738540 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc980070200 con 0x7fc98006f460 2024-03-25T18:30:18.144 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.142+0000 7fc985215640 10 monclient: _finish_auth 0 2024-03-25T18:30:18.144 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.142+0000 7fc985215640 10 monclient: _check_auth_tickets 2024-03-25T18:30:18.145 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.142+0000 7fc985215640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:18.145 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.142+0000 7fc985215640 1 -- 172.21.15.3:0/4116738540 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc97c001690 con 0x7fc98006f460 2024-03-25T18:30:18.145 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.142+0000 7fc985215640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:18.145 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.142+0000 7fc985215640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:18.145 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.142+0000 7fc985215640 10 monclient: dump: 2024-03-25T18:30:18.145 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:18.145 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:18.145 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:18.145 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:18.145 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:18.145 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:18.145 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:18.145 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:18.145 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:18.145 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:18.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.142+0000 7fc98847b640 5 monclient: authenticate success, global_id 4426 2024-03-25T18:30:18.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.142+0000 7fc98847b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:18.149 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.148+0000 7fc985215640 1 -- 172.21.15.3:0/4116738540 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc97c0017f0 con 0x7fc98006f460 2024-03-25T18:30:18.149 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.148+0000 7fc985215640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:18.149 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.148+0000 7fc985215640 1 -- 172.21.15.3:0/4116738540 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc97c001b80 con 0x7fc98006f460 2024-03-25T18:30:18.150 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.148+0000 7fc98847b640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:18.150 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.148+0000 7fc98847b640 4 set_mon_vals no callback set 2024-03-25T18:30:18.150 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.148+0000 7fc985215640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:18.150 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.148+0000 7fc985215640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:18.150 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.148+0000 7fc985215640 10 monclient: dump: 2024-03-25T18:30:18.150 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:18.150 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:18.150 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:18.150 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:18.150 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:18.150 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:18.150 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:18.150 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:18.151 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:18.151 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:18.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.149+0000 7fc98847b640 10 monclient: shutdown 2024-03-25T18:30:18.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.149+0000 7fc98847b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:18.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.149+0000 7fc98847b640 1 -- 172.21.15.3:0/4116738540 >> v1:172.21.15.16:6789/0 conn(0x7fc98006f460 legacy=0x7fc980142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:18.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.149+0000 7fc98847b640 1 -- 172.21.15.3:0/4116738540 shutdown_connections 2024-03-25T18:30:18.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.149+0000 7fc98847b640 1 -- 172.21.15.3:0/4116738540 >> 172.21.15.3:0/4116738540 conn(0x7fc9800b4bd0 msgr2=0x7fc9800b4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:18.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.149+0000 7fc98847b640 1 -- 172.21.15.3:0/4116738540 shutdown_connections 2024-03-25T18:30:18.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.149+0000 7fc98847b640 1 -- 172.21.15.3:0/4116738540 wait complete. 2024-03-25T18:30:18.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.150+0000 7fc98847b640 10 monclient: build_initial_monmap 2024-03-25T18:30:18.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.150+0000 7fc98847b640 10 monclient: monmap: 2024-03-25T18:30:18.153 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:30:18.153 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:18.153 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:30:18.153 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:30:18.153 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:18.154 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:30:18.154 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:30:18.154 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:30:18.154 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:30:18.154 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:18.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.150+0000 7fc98847b640 1 librados: starting msgr at 2024-03-25T18:30:18.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.150+0000 7fc98847b640 1 librados: starting objecter 2024-03-25T18:30:18.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.150+0000 7fc98847b640 1 Processor -- start 2024-03-25T18:30:18.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.150+0000 7fc98847b640 1 -- start start 2024-03-25T18:30:18.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.150+0000 7fc98847b640 1 librados: setting wanted keys 2024-03-25T18:30:18.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.150+0000 7fc98847b640 1 librados: calling monclient init 2024-03-25T18:30:18.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.150+0000 7fc98847b640 10 monclient: init 2024-03-25T18:30:18.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.151+0000 7fc98847b640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:18.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.151+0000 7fc98847b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:30:18.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.151+0000 7fc98847b640 10 monclient(hunting): picked mon.noname-b con 0x7fc98006f460 addr v1:172.21.15.16:6789/0 2024-03-25T18:30:18.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.151+0000 7fc98847b640 10 monclient(hunting): picked mon.noname-a con 0x7fc98006f840 addr v1:172.21.15.3:6789/0 2024-03-25T18:30:18.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.151+0000 7fc98847b640 10 monclient(hunting): picked mon.noname-c con 0x7fc9800728f0 addr v1:172.21.15.3:6790/0 2024-03-25T18:30:18.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.151+0000 7fc98847b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9800ad290 con 0x7fc98006f840 2024-03-25T18:30:18.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.151+0000 7fc98847b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9800aae80 con 0x7fc98006f460 2024-03-25T18:30:18.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.151+0000 7fc98847b640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9800adf10 con 0x7fc9800728f0 2024-03-25T18:30:18.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.151+0000 7fc98847b640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:18.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.151+0000 7fc98847b640 10 monclient(hunting): authenticate will time out at 7672.411999s 2024-03-25T18:30:18.161 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.161+0000 7fc986217640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc98006f460 0x7fc9800a94e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:52866/0 (socket says 172.21.15.3:52866) 2024-03-25T18:30:18.162 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.161+0000 7fc986217640 1 -- 172.21.15.3:0/2777398623 learned_addr learned my addr 172.21.15.3:0/2777398623 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:30:18.162 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.161+0000 7fc976ffd640 10 client.?.objecter ms_handle_connect 0x7fc98006f460 2024-03-25T18:30:18.162 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.161+0000 7fc976ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:18.172 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.171+0000 7fc976ffd640 1 -- 172.21.15.3:0/2777398623 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2656749764 0 0) 0x7fc9800aae80 con 0x7fc98006f460 2024-03-25T18:30:18.172 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.171+0000 7fc976ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:18.172 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.171+0000 7fc976ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:18.172 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.171+0000 7fc976ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:18.172 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.171+0000 7fc976ffd640 10 monclient(hunting): my global_id is 4429 2024-03-25T18:30:18.173 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.171+0000 7fc976ffd640 1 -- 172.21.15.3:0/2777398623 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc964004eb0 con 0x7fc98006f460 2024-03-25T18:30:18.179 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.178+0000 7fc976ffd640 10 client.?.objecter ms_handle_connect 0x7fc9800728f0 2024-03-25T18:30:18.179 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.178+0000 7fc976ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:18.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.183+0000 7fc976ffd640 1 -- 172.21.15.3:0/2777398623 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2611989048 0 0) 0x7fc964004eb0 con 0x7fc98006f460 2024-03-25T18:30:18.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.183+0000 7fc976ffd640 1 -- 172.21.15.3:0/2777398623 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9800aae80 con 0x7fc98006f460 2024-03-25T18:30:18.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.183+0000 7fc976ffd640 1 -- 172.21.15.3:0/2777398623 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc97c001b00 con 0x7fc98006f460 2024-03-25T18:30:18.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.183+0000 7fc976ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:18.185 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.183+0000 7fc976ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:18.185 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.183+0000 7fc976ffd640 10 monclient(hunting): dump: 2024-03-25T18:30:18.185 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:18.185 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:18.185 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:18.185 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:18.185 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:18.185 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:18.185 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:18.185 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:18.185 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:18.185 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:18.185 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.183+0000 7fc976ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:18.187 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.186+0000 7fc976ffd640 10 client.?.objecter ms_handle_connect 0x7fc98006f840 2024-03-25T18:30:18.187 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.186+0000 7fc976ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:18.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.191+0000 7fc976ffd640 1 -- 172.21.15.3:0/2777398623 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3068986883 0 0) 0x7fc9800aae80 con 0x7fc98006f460 2024-03-25T18:30:18.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.191+0000 7fc976ffd640 1 -- 172.21.15.3:0/2777398623 >> v1:172.21.15.3:6790/0 conn(0x7fc9800728f0 legacy=0x7fc9801defe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:18.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.191+0000 7fc976ffd640 1 -- 172.21.15.3:0/2777398623 >> v1:172.21.15.3:6789/0 conn(0x7fc98006f840 legacy=0x7fc9800a9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:18.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.191+0000 7fc976ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:18.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.191+0000 7fc976ffd640 1 monclient: found mon.b 2024-03-25T18:30:18.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.191+0000 7fc976ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:18.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.192+0000 7fc976ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:30:18.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.192+0000 7fc976ffd640 1 -- 172.21.15.3:0/2777398623 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9800ae0a0 con 0x7fc98006f460 2024-03-25T18:30:18.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.192+0000 7fc976ffd640 10 monclient: _finish_auth 0 2024-03-25T18:30:18.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.192+0000 7fc976ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:18.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.192+0000 7fc976ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:18.195 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.192+0000 7fc98847b640 5 monclient: authenticate success, global_id 4429 2024-03-25T18:30:18.195 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.192+0000 7fc98847b640 10 monclient: _renew_subs 2024-03-25T18:30:18.195 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.192+0000 7fc98847b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:30:18.195 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.192+0000 7fc98847b640 1 -- 172.21.15.3:0/2777398623 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9800ac200 con 0x7fc98006f460 2024-03-25T18:30:18.195 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.192+0000 7fc98847b640 10 client.4429.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:18.195 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.192+0000 7fc98847b640 10 monclient: _renew_subs 2024-03-25T18:30:18.195 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.192+0000 7fc98847b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:30:18.195 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.192+0000 7fc98847b640 1 -- 172.21.15.3:0/2777398623 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc98013f660 con 0x7fc98006f460 2024-03-25T18:30:18.195 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.192+0000 7fc98847b640 1 librados: init done 2024-03-25T18:30:18.195 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.193+0000 7fc98847b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:18.195 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.193+0000 7fc98847b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:18.195 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.193+0000 7fc98847b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:30:18.195 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.193+0000 7fc98847b640 1 -- 172.21.15.3:0/2777398623 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc980072d20 con 0x7fc98006f460 2024-03-25T18:30:18.490 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.489+0000 7fc976ffd640 1 -- 172.21.15.3:0/2777398623 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc97c004a90 con 0x7fc98006f460 2024-03-25T18:30:18.491 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.489+0000 7fc976ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:18.491 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.489+0000 7fc976ffd640 1 -- 172.21.15.3:0/2777398623 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc97c004ea0 con 0x7fc98006f460 2024-03-25T18:30:18.491 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.489+0000 7fc976ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:18.491 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.489+0000 7fc976ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:18.491 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.489+0000 7fc976ffd640 10 monclient: dump: 2024-03-25T18:30:18.491 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:18.491 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:18.491 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:18.491 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:18.491 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:18.491 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:18.491 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:18.491 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:18.491 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:18.492 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:18.492 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.489+0000 7fc984a14640 4 set_mon_vals no callback set 2024-03-25T18:30:18.492 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.490+0000 7fc976ffd640 1 -- 172.21.15.3:0/2777398623 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7fc97c0050c0 con 0x7fc98006f460 2024-03-25T18:30:18.492 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.490+0000 7fc985a16640 10 monclient: get_auth_request con 0x7fc96406dce0 auth_method 0 2024-03-25T18:30:18.492 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.491+0000 7fc976ffd640 10 client.4429.objecter ms_handle_connect 0x7fc96406dce0 2024-03-25T18:30:18.495 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.494+0000 7fc976ffd640 1 -- 172.21.15.3:0/2777398623 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7fc97c054940 con 0x7fc98006f460 2024-03-25T18:30:18.495 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.494+0000 7fc976ffd640 10 client.4429.objecter ms_dispatch 0x7fc980000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:18.496 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.494+0000 7fc976ffd640 3 client.4429.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:18.496 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.494+0000 7fc976ffd640 3 client.4429.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:18.496 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.494+0000 7fc976ffd640 20 client.4429.objecter dump_active .. 0 homeless 2024-03-25T18:30:18.496 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.495+0000 7fc976ffd640 1 -- 172.21.15.3:0/2777398623 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc97c050450 con 0x7fc98006f460 2024-03-25T18:30:18.496 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.495+0000 7fc976ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:18.496 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.495+0000 7fc976ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:18.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.755+0000 7fc98847b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:18.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.755+0000 7fc98847b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:18.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.755+0000 7fc98847b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:30:18.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.755+0000 7fc98847b640 1 -- 172.21.15.3:0/2777398623 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9800b3ac0 con 0x7fc98006f460 2024-03-25T18:30:18.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.856+0000 7fc976ffd640 1 -- 172.21.15.3:0/2777398623 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7fc97c052380 con 0x7fc98006f460 2024-03-25T18:30:18.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.856+0000 7fc976ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:18.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.856+0000 7fc976ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:18.857 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:30:18.857 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:18.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.858+0000 7fc98847b640 10 librados: watch_flush enter 2024-03-25T18:30:18.860 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.858+0000 7fc98847b640 10 librados: watch_flush exit 2024-03-25T18:30:18.860 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.858+0000 7fc98847b640 20 client.4429.objecter shutdown clearing up homeless session... 2024-03-25T18:30:18.860 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.858+0000 7fc98847b640 10 client.4429.objecter successfully canceled tick 2024-03-25T18:30:18.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.858+0000 7fc98847b640 1 -- 172.21.15.3:0/2777398623 >> v1:172.21.15.3:6800/4195140966 conn(0x7fc96406dce0 legacy=0x7fc9640701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:18.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.858+0000 7fc98847b640 10 monclient: shutdown 2024-03-25T18:30:18.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.858+0000 7fc98847b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:18.862 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.858+0000 7fc98847b640 1 -- 172.21.15.3:0/2777398623 >> v1:172.21.15.16:6789/0 conn(0x7fc98006f460 legacy=0x7fc9800a94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:18.862 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.858+0000 7fc98847b640 1 -- 172.21.15.3:0/2777398623 shutdown_connections 2024-03-25T18:30:18.862 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.858+0000 7fc98847b640 1 -- 172.21.15.3:0/2777398623 >> 172.21.15.3:0/2777398623 conn(0x7fc9800b4bd0 msgr2=0x7fc9800ae8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:18.862 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.858+0000 7fc98847b640 1 -- 172.21.15.3:0/2777398623 shutdown_connections 2024-03-25T18:30:18.862 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.858+0000 7fc98847b640 1 -- 172.21.15.3:0/2777398623 wait complete. 2024-03-25T18:30:18.862 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:18.858+0000 7fc98847b640 1 librados: shutdown 2024-03-25T18:30:18.871 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:21.930 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:05:35.658 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:21.940 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:05:35.670 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:21.984 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:05:35.713 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:21.992 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:05:35.683 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:21.998 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:05:35.690 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:22.006 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:05:35.735 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:22.016 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:05:35.745 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:22.026 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:05:35.718 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:22.037 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:05:35.766 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:25.678 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:25.792 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.791+0000 7fe44ea4a640 10 monclient: get_monmap_and_config 2024-03-25T18:30:25.792 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.791+0000 7fe44ea4a640 10 monclient: build_initial_monmap 2024-03-25T18:30:25.792 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.791+0000 7fe44ea4a640 10 monclient: monmap: 2024-03-25T18:30:25.793 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:30:25.793 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:25.793 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:30:25.792568+0000 2024-03-25T18:30:25.793 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:30:25.792568+0000 2024-03-25T18:30:25.793 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:25.793 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:25.793 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:30:25.793 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:30:25.793 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:30:25.793 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:25.795 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.792+0000 7fe44ea4a640 1 Processor -- start 2024-03-25T18:30:25.795 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.792+0000 7fe44ea4a640 1 -- start start 2024-03-25T18:30:25.795 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.792+0000 7fe44ea4a640 10 monclient: init 2024-03-25T18:30:25.795 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.792+0000 7fe44ea4a640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:25.795 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.792+0000 7fe44ea4a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:30:25.795 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.792+0000 7fe44ea4a640 10 monclient(hunting): picked mon.noname-a con 0x7fe4480b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:30:25.795 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.792+0000 7fe44ea4a640 10 monclient(hunting): picked mon.noname-c con 0x7fe4480b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:30:25.795 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.792+0000 7fe44ea4a640 10 monclient(hunting): picked mon.noname-b con 0x7fe44813c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:30:25.795 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.792+0000 7fe44ea4a640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4480ab0e0 con 0x7fe4480b4ff0 2024-03-25T18:30:25.795 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.793+0000 7fe44ea4a640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4480ab3c0 con 0x7fe4480b4670 2024-03-25T18:30:25.796 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.793+0000 7fe44ea4a640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4480a9f30 con 0x7fe44813c480 2024-03-25T18:30:25.796 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.793+0000 7fe44ea4a640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:25.796 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.793+0000 7fe44ea4a640 10 monclient(hunting): authenticate will time out at 7680.053564s 2024-03-25T18:30:25.796 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.793+0000 7fe4477fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe4480b4670 0x7fe4481425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:58080/0 (socket says 172.21.15.3:58080) 2024-03-25T18:30:25.796 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.793+0000 7fe4477fe640 1 -- 172.21.15.3:0/3190612242 learned_addr learned my addr 172.21.15.3:0/3190612242 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:30:25.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.866+0000 7fe446ffd640 1 -- 172.21.15.3:0/3190612242 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1512255477 0 0) 0x7fe4480ab0e0 con 0x7fe4480b4ff0 2024-03-25T18:30:25.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.866+0000 7fe446ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:25.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.866+0000 7fe446ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:25.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.866+0000 7fe446ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:25.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.866+0000 7fe446ffd640 10 monclient(hunting): my global_id is 4473 2024-03-25T18:30:25.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.866+0000 7fe446ffd640 1 -- 172.21.15.3:0/3190612242 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe430003520 con 0x7fe4480b4ff0 2024-03-25T18:30:25.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.874+0000 7fe446ffd640 1 -- 172.21.15.3:0/3190612242 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2107219156 0 0) 0x7fe430003520 con 0x7fe4480b4ff0 2024-03-25T18:30:25.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.874+0000 7fe446ffd640 1 -- 172.21.15.3:0/3190612242 >> v1:172.21.15.3:6790/0 conn(0x7fe44813c480 legacy=0x7fe44814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:25.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.874+0000 7fe446ffd640 1 -- 172.21.15.3:0/3190612242 >> v1:172.21.15.16:6789/0 conn(0x7fe4480b4670 legacy=0x7fe4481425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:25.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.874+0000 7fe446ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:25.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.874+0000 7fe446ffd640 1 monclient: found mon.noname-a 2024-03-25T18:30:25.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.874+0000 7fe446ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:25.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.874+0000 7fe446ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:30:25.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.874+0000 7fe446ffd640 1 -- 172.21.15.3:0/3190612242 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4480a9b70 con 0x7fe4480b4ff0 2024-03-25T18:30:25.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.874+0000 7fe446ffd640 10 monclient: _finish_auth 0 2024-03-25T18:30:25.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.874+0000 7fe446ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:25.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.874+0000 7fe446ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:25.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.874+0000 7fe446ffd640 1 -- 172.21.15.3:0/3190612242 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fe438001690 con 0x7fe4480b4ff0 2024-03-25T18:30:25.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.874+0000 7fe446ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:25.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.874+0000 7fe446ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:25.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.874+0000 7fe446ffd640 10 monclient: dump: 2024-03-25T18:30:25.877 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:25.877 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:25.877 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:25.877 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:25.877 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:25.877 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:25.877 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:25.877 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:25.877 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:25.877 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:25.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.874+0000 7fe44ea4a640 5 monclient: authenticate success, global_id 4473 2024-03-25T18:30:25.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.874+0000 7fe44ea4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:25.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.888+0000 7fe446ffd640 1 -- 172.21.15.3:0/3190612242 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4380017f0 con 0x7fe4480b4ff0 2024-03-25T18:30:25.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.888+0000 7fe446ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:25.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.888+0000 7fe446ffd640 1 -- 172.21.15.3:0/3190612242 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fe438001c00 con 0x7fe4480b4ff0 2024-03-25T18:30:25.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.888+0000 7fe44ea4a640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:25.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.888+0000 7fe44ea4a640 4 set_mon_vals no callback set 2024-03-25T18:30:25.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.888+0000 7fe446ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:25.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.888+0000 7fe446ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:25.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.888+0000 7fe446ffd640 10 monclient: dump: 2024-03-25T18:30:25.890 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:25.890 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:25.890 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:25.890 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:25.890 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:25.890 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:25.890 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:25.890 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:25.890 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:25.890 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:25.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.889+0000 7fe44ea4a640 10 monclient: shutdown 2024-03-25T18:30:25.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.889+0000 7fe44ea4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:25.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.889+0000 7fe44ea4a640 1 -- 172.21.15.3:0/3190612242 >> v1:172.21.15.3:6789/0 conn(0x7fe4480b4ff0 legacy=0x7fe44813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:25.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.889+0000 7fe44ea4a640 1 -- 172.21.15.3:0/3190612242 shutdown_connections 2024-03-25T18:30:25.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.889+0000 7fe44ea4a640 1 -- 172.21.15.3:0/3190612242 >> 172.21.15.3:0/3190612242 conn(0x7fe4480617a0 msgr2=0x7fe448061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:25.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.889+0000 7fe44ea4a640 1 -- 172.21.15.3:0/3190612242 shutdown_connections 2024-03-25T18:30:25.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.889+0000 7fe44ea4a640 1 -- 172.21.15.3:0/3190612242 wait complete. 2024-03-25T18:30:25.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.889+0000 7fe44ea4a640 10 monclient: build_initial_monmap 2024-03-25T18:30:25.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.889+0000 7fe44ea4a640 10 monclient: monmap: 2024-03-25T18:30:25.893 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:30:25.893 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:25.893 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:30:25.893 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:30:25.893 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:25.893 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:30:25.894 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:30:25.894 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:30:25.894 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:30:25.894 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:25.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.890+0000 7fe44ea4a640 1 librados: starting msgr at 2024-03-25T18:30:25.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.890+0000 7fe44ea4a640 1 librados: starting objecter 2024-03-25T18:30:25.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.890+0000 7fe44ea4a640 1 Processor -- start 2024-03-25T18:30:25.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.890+0000 7fe44ea4a640 1 -- start start 2024-03-25T18:30:25.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.890+0000 7fe44ea4a640 1 librados: setting wanted keys 2024-03-25T18:30:25.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.890+0000 7fe44ea4a640 1 librados: calling monclient init 2024-03-25T18:30:25.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.890+0000 7fe44ea4a640 10 monclient: init 2024-03-25T18:30:25.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.890+0000 7fe44ea4a640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:25.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.890+0000 7fe44ea4a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:30:25.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.890+0000 7fe44ea4a640 10 monclient(hunting): picked mon.noname-c con 0x7fe4480b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:30:25.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.890+0000 7fe44ea4a640 10 monclient(hunting): picked mon.noname-a con 0x7fe4480b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:30:25.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.891+0000 7fe44ea4a640 10 monclient(hunting): picked mon.noname-b con 0x7fe44813c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:30:25.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.891+0000 7fe44ea4a640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4480b0b90 con 0x7fe4480b4ff0 2024-03-25T18:30:25.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.891+0000 7fe44ea4a640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4480ae6d0 con 0x7fe44813c480 2024-03-25T18:30:25.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.891+0000 7fe44ea4a640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe44814e1f0 con 0x7fe4480b4670 2024-03-25T18:30:25.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.891+0000 7fe44ea4a640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:25.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.891+0000 7fe44ea4a640 10 monclient(hunting): authenticate will time out at 7680.151495s 2024-03-25T18:30:25.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.905+0000 7fe4477fe640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fe4480b4ff0 0x7fe4480aebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:33250/0 (socket says 172.21.15.3:33250) 2024-03-25T18:30:25.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.905+0000 7fe4477fe640 1 -- 172.21.15.3:0/2068773883 learned_addr learned my addr 172.21.15.3:0/2068773883 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:30:25.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.905+0000 7fe444ff9640 10 client.?.objecter ms_handle_connect 0x7fe4480b4ff0 2024-03-25T18:30:25.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.905+0000 7fe444ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:25.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.916+0000 7fe444ff9640 1 -- 172.21.15.3:0/2068773883 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2326342406 0 0) 0x7fe4480b0b90 con 0x7fe4480b4ff0 2024-03-25T18:30:25.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.916+0000 7fe444ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:25.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.916+0000 7fe444ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:25.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.916+0000 7fe444ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:25.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.916+0000 7fe444ff9640 10 monclient(hunting): my global_id is 4476 2024-03-25T18:30:25.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.916+0000 7fe444ff9640 1 -- 172.21.15.3:0/2068773883 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe434004eb0 con 0x7fe4480b4ff0 2024-03-25T18:30:25.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.925+0000 7fe444ff9640 1 -- 172.21.15.3:0/2068773883 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2199583882 0 0) 0x7fe434004eb0 con 0x7fe4480b4ff0 2024-03-25T18:30:25.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.925+0000 7fe444ff9640 1 -- 172.21.15.3:0/2068773883 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4480b0b90 con 0x7fe4480b4ff0 2024-03-25T18:30:25.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.925+0000 7fe444ff9640 1 -- 172.21.15.3:0/2068773883 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fe43c0018b0 con 0x7fe4480b4ff0 2024-03-25T18:30:25.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.925+0000 7fe444ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:25.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.925+0000 7fe444ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:25.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.925+0000 7fe444ff9640 10 monclient(hunting): dump: 2024-03-25T18:30:25.927 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:25.927 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:25.927 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:25.928 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:25.928 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:25.928 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:25.928 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:25.928 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:25.928 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:25.928 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:25.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.925+0000 7fe444ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:25.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.927+0000 7fe444ff9640 10 client.?.objecter ms_handle_connect 0x7fe44813c480 2024-03-25T18:30:25.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.927+0000 7fe444ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:25.944 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.942+0000 7fe444ff9640 1 -- 172.21.15.3:0/2068773883 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 899237240 0 0) 0x7fe4480b0b90 con 0x7fe4480b4ff0 2024-03-25T18:30:25.944 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.942+0000 7fe444ff9640 1 -- 172.21.15.3:0/2068773883 >> v1:172.21.15.3:6790/0 conn(0x7fe4480b4670 legacy=0x7fe4480b1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:25.944 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.942+0000 7fe444ff9640 1 -- 172.21.15.3:0/2068773883 >> v1:172.21.15.16:6789/0 conn(0x7fe44813c480 legacy=0x7fe4480af300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:25.944 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.942+0000 7fe444ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:30:25.944 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.942+0000 7fe444ff9640 1 monclient: found mon.a 2024-03-25T18:30:25.944 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.942+0000 7fe444ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:25.944 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.942+0000 7fe444ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:30:25.945 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.942+0000 7fe444ff9640 1 -- 172.21.15.3:0/2068773883 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe44814e380 con 0x7fe4480b4ff0 2024-03-25T18:30:25.945 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.942+0000 7fe444ff9640 10 monclient: _finish_auth 0 2024-03-25T18:30:25.945 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.942+0000 7fe444ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:30:25.945 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.942+0000 7fe444ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:25.945 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.942+0000 7fe44ea4a640 5 monclient: authenticate success, global_id 4476 2024-03-25T18:30:25.945 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.942+0000 7fe44ea4a640 10 monclient: _renew_subs 2024-03-25T18:30:25.945 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.942+0000 7fe44ea4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:30:25.945 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.942+0000 7fe44ea4a640 1 -- 172.21.15.3:0/2068773883 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4481e37f0 con 0x7fe4480b4ff0 2024-03-25T18:30:25.945 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.942+0000 7fe44ea4a640 10 client.4476.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:25.945 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.942+0000 7fe44ea4a640 10 monclient: _renew_subs 2024-03-25T18:30:25.945 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.942+0000 7fe44ea4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:30:25.945 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.942+0000 7fe44ea4a640 1 -- 172.21.15.3:0/2068773883 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4481e3b60 con 0x7fe4480b4ff0 2024-03-25T18:30:25.945 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.942+0000 7fe44ea4a640 1 librados: init done 2024-03-25T18:30:25.946 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.943+0000 7fe44ea4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:25.946 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.943+0000 7fe44ea4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:25.946 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.943+0000 7fe44ea4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:30:25.946 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:25.943+0000 7fe44ea4a640 1 -- 172.21.15.3:0/2068773883 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4480ae530 con 0x7fe4480b4ff0 2024-03-25T18:30:26.163 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.162+0000 7fe444ff9640 1 -- 172.21.15.3:0/2068773883 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe43c001a10 con 0x7fe4480b4ff0 2024-03-25T18:30:26.163 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.162+0000 7fe444ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:26.163 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.162+0000 7fe4467fc640 4 set_mon_vals no callback set 2024-03-25T18:30:26.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.203+0000 7fe444ff9640 1 -- 172.21.15.3:0/2068773883 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fe43c0048e0 con 0x7fe4480b4ff0 2024-03-25T18:30:26.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.203+0000 7fe444ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:26.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.203+0000 7fe444ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:26.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.203+0000 7fe444ff9640 10 monclient: dump: 2024-03-25T18:30:26.204 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:26.204 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:26.204 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:26.204 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:26.205 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:26.205 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:26.205 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:26.205 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:26.205 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:26.205 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:26.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.206+0000 7fe444ff9640 1 -- 172.21.15.3:0/2068773883 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7fe43c004ac0 con 0x7fe4480b4ff0 2024-03-25T18:30:26.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.206+0000 7fe447fff640 10 monclient: get_auth_request con 0x7fe43406dbc0 auth_method 0 2024-03-25T18:30:26.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.207+0000 7fe444ff9640 10 client.4476.objecter ms_handle_connect 0x7fe43406dbc0 2024-03-25T18:30:26.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.212+0000 7fe444ff9640 1 -- 172.21.15.3:0/2068773883 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7fe43c054800 con 0x7fe4480b4ff0 2024-03-25T18:30:26.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.212+0000 7fe444ff9640 10 client.4476.objecter ms_dispatch 0x7fe448000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:26.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.212+0000 7fe444ff9640 3 client.4476.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:26.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.212+0000 7fe444ff9640 3 client.4476.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:26.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.212+0000 7fe444ff9640 20 client.4476.objecter dump_active .. 0 homeless 2024-03-25T18:30:26.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.212+0000 7fe444ff9640 1 -- 172.21.15.3:0/2068773883 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe43c050140 con 0x7fe4480b4ff0 2024-03-25T18:30:26.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.212+0000 7fe444ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:26.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.212+0000 7fe444ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:26.461 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.460+0000 7fe44ea4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:26.461 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.460+0000 7fe44ea4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:26.461 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.460+0000 7fe44ea4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:30:26.462 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.460+0000 7fe44ea4a640 1 -- 172.21.15.3:0/2068773883 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe44806dbd0 con 0x7fe4480b4ff0 2024-03-25T18:30:26.486 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.485+0000 7fe444ff9640 1 -- 172.21.15.3:0/2068773883 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7fe43c052240 con 0x7fe4480b4ff0 2024-03-25T18:30:26.486 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.485+0000 7fe444ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:26.486 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.485+0000 7fe444ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:26.486 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:30:26.486 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:26.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.487+0000 7fe44ea4a640 10 librados: watch_flush enter 2024-03-25T18:30:26.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.487+0000 7fe44ea4a640 10 librados: watch_flush exit 2024-03-25T18:30:26.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.487+0000 7fe44ea4a640 20 client.4476.objecter shutdown clearing up homeless session... 2024-03-25T18:30:26.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.487+0000 7fe44ea4a640 10 client.4476.objecter successfully canceled tick 2024-03-25T18:30:26.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.487+0000 7fe44ea4a640 1 -- 172.21.15.3:0/2068773883 >> v1:172.21.15.3:6800/4195140966 conn(0x7fe43406dbc0 legacy=0x7fe434070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:26.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.487+0000 7fe44ea4a640 10 monclient: shutdown 2024-03-25T18:30:26.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.487+0000 7fe44ea4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:26.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.487+0000 7fe44ea4a640 1 -- 172.21.15.3:0/2068773883 >> v1:172.21.15.3:6789/0 conn(0x7fe4480b4ff0 legacy=0x7fe4480aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:26.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.487+0000 7fe44ea4a640 1 -- 172.21.15.3:0/2068773883 shutdown_connections 2024-03-25T18:30:26.490 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.487+0000 7fe44ea4a640 1 -- 172.21.15.3:0/2068773883 >> 172.21.15.3:0/2068773883 conn(0x7fe4480617a0 msgr2=0x7fe4481407a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:26.490 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.487+0000 7fe44ea4a640 1 -- 172.21.15.3:0/2068773883 shutdown_connections 2024-03-25T18:30:26.490 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.488+0000 7fe44ea4a640 1 -- 172.21.15.3:0/2068773883 wait complete. 2024-03-25T18:30:26.490 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:26.488+0000 7fe44ea4a640 1 librados: shutdown 2024-03-25T18:30:26.501 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:33.309 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:33.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.423+0000 7fa0ff3f6640 10 monclient: get_monmap_and_config 2024-03-25T18:30:33.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.423+0000 7fa0ff3f6640 10 monclient: build_initial_monmap 2024-03-25T18:30:33.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.423+0000 7fa0ff3f6640 10 monclient: monmap: 2024-03-25T18:30:33.425 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:30:33.425 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:33.425 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:30:33.424399+0000 2024-03-25T18:30:33.425 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:30:33.424399+0000 2024-03-25T18:30:33.425 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:33.425 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:33.425 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:30:33.425 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:30:33.426 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:30:33.426 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:33.426 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.423+0000 7fa0ff3f6640 1 Processor -- start 2024-03-25T18:30:33.426 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.424+0000 7fa0ff3f6640 1 -- start start 2024-03-25T18:30:33.426 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.424+0000 7fa0ff3f6640 10 monclient: init 2024-03-25T18:30:33.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.424+0000 7fa0ff3f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:33.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.424+0000 7fa0ff3f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:30:33.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.424+0000 7fa0ff3f6640 10 monclient(hunting): picked mon.noname-a con 0x7fa0f80b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:30:33.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.424+0000 7fa0ff3f6640 10 monclient(hunting): picked mon.noname-c con 0x7fa0f80b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:30:33.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.425+0000 7fa0ff3f6640 10 monclient(hunting): picked mon.noname-b con 0x7fa0f813c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:30:33.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.425+0000 7fa0ff3f6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f80ab0e0 con 0x7fa0f80b4ff0 2024-03-25T18:30:33.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.425+0000 7fa0ff3f6640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f80ab3c0 con 0x7fa0f80b4670 2024-03-25T18:30:33.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.425+0000 7fa0ff3f6640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f80a9f30 con 0x7fa0f813c480 2024-03-25T18:30:33.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.425+0000 7fa0ff3f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:33.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.425+0000 7fa0ff3f6640 10 monclient(hunting): authenticate will time out at 7687.685600s 2024-03-25T18:30:33.436 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.435+0000 7fa0fc991640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa0f80b4670 0x7fa0f81425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:38418/0 (socket says 172.21.15.3:38418) 2024-03-25T18:30:33.436 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.435+0000 7fa0fc991640 1 -- 172.21.15.3:0/1013815826 learned_addr learned my addr 172.21.15.3:0/1013815826 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:30:33.454 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.453+0000 7fa0effff640 1 -- 172.21.15.3:0/1013815826 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3548565357 0 0) 0x7fa0f80ab3c0 con 0x7fa0f80b4670 2024-03-25T18:30:33.454 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.453+0000 7fa0effff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:33.454 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.453+0000 7fa0effff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:33.454 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.453+0000 7fa0effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:33.454 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.453+0000 7fa0effff640 10 monclient(hunting): my global_id is 4432 2024-03-25T18:30:33.455 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.453+0000 7fa0effff640 1 -- 172.21.15.3:0/1013815826 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0e4003520 con 0x7fa0f80b4670 2024-03-25T18:30:33.458 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.457+0000 7fa0effff640 1 -- 172.21.15.3:0/1013815826 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1293244684 0 0) 0x7fa0f80ab0e0 con 0x7fa0f80b4ff0 2024-03-25T18:30:33.459 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.457+0000 7fa0effff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:33.459 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.457+0000 7fa0effff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:33.459 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.457+0000 7fa0effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:33.459 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.457+0000 7fa0effff640 10 monclient(hunting): my global_id is 4479 2024-03-25T18:30:33.459 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.458+0000 7fa0effff640 1 -- 172.21.15.3:0/1013815826 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0f80ab3c0 con 0x7fa0f80b4ff0 2024-03-25T18:30:33.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.463+0000 7fa0effff640 1 -- 172.21.15.3:0/1013815826 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1501361757 0 0) 0x7fa0e4003520 con 0x7fa0f80b4670 2024-03-25T18:30:33.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.463+0000 7fa0effff640 1 -- 172.21.15.3:0/1013815826 >> v1:172.21.15.3:6790/0 conn(0x7fa0f813c480 legacy=0x7fa0f814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:33.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.463+0000 7fa0effff640 1 -- 172.21.15.3:0/1013815826 >> v1:172.21.15.3:6789/0 conn(0x7fa0f80b4ff0 legacy=0x7fa0f813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:33.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.463+0000 7fa0effff640 10 monclient: _finish_hunting 0 2024-03-25T18:30:33.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.463+0000 7fa0effff640 1 monclient: found mon.noname-c 2024-03-25T18:30:33.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.463+0000 7fa0effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:33.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.463+0000 7fa0effff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:30:33.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.463+0000 7fa0effff640 1 -- 172.21.15.3:0/1013815826 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f80a9b70 con 0x7fa0f80b4670 2024-03-25T18:30:33.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.463+0000 7fa0effff640 10 monclient: _finish_auth 0 2024-03-25T18:30:33.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.463+0000 7fa0effff640 10 monclient: _check_auth_tickets 2024-03-25T18:30:33.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.463+0000 7fa0effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:33.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.463+0000 7fa0effff640 1 -- 172.21.15.3:0/1013815826 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fa0e8001690 con 0x7fa0f80b4670 2024-03-25T18:30:33.466 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.463+0000 7fa0ff3f6640 5 monclient: authenticate success, global_id 4432 2024-03-25T18:30:33.466 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.463+0000 7fa0ff3f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:33.466 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.463+0000 7fa0effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:33.466 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.463+0000 7fa0effff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:33.466 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.463+0000 7fa0effff640 10 monclient: dump: 2024-03-25T18:30:33.466 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:33.466 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:33.466 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:33.466 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:33.466 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:33.466 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:33.466 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:33.466 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:33.466 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:33.467 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:33.467 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.463+0000 7fa0ff3f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:33.482 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.481+0000 7fa0effff640 1 -- 172.21.15.3:0/1013815826 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0e80017f0 con 0x7fa0f80b4670 2024-03-25T18:30:33.482 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.481+0000 7fa0effff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:33.482 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.481+0000 7fa0effff640 1 -- 172.21.15.3:0/1013815826 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fa0e8001c00 con 0x7fa0f80b4670 2024-03-25T18:30:33.482 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.481+0000 7fa0effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:33.482 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.481+0000 7fa0effff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:33.482 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.481+0000 7fa0effff640 10 monclient: dump: 2024-03-25T18:30:33.482 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:33.482 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:33.482 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:33.483 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:33.483 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:33.483 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:33.483 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:33.483 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:33.483 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:33.483 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:33.483 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.481+0000 7fa0ff3f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:33.483 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.481+0000 7fa0ff3f6640 4 set_mon_vals no callback set 2024-03-25T18:30:33.485 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.482+0000 7fa0ff3f6640 10 monclient: shutdown 2024-03-25T18:30:33.485 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.482+0000 7fa0ff3f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:33.485 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.482+0000 7fa0ff3f6640 1 -- 172.21.15.3:0/1013815826 >> v1:172.21.15.16:6789/0 conn(0x7fa0f80b4670 legacy=0x7fa0f81425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:33.485 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.482+0000 7fa0ff3f6640 1 -- 172.21.15.3:0/1013815826 shutdown_connections 2024-03-25T18:30:33.485 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.482+0000 7fa0ff3f6640 1 -- 172.21.15.3:0/1013815826 >> 172.21.15.3:0/1013815826 conn(0x7fa0f80617a0 msgr2=0x7fa0f8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:33.485 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.482+0000 7fa0ff3f6640 1 -- 172.21.15.3:0/1013815826 shutdown_connections 2024-03-25T18:30:33.485 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.482+0000 7fa0ff3f6640 1 -- 172.21.15.3:0/1013815826 wait complete. 2024-03-25T18:30:33.485 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.483+0000 7fa0ff3f6640 10 monclient: build_initial_monmap 2024-03-25T18:30:33.485 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.483+0000 7fa0ff3f6640 10 monclient: monmap: 2024-03-25T18:30:33.485 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:30:33.486 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:33.486 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:30:33.486 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:30:33.486 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:33.486 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:30:33.486 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:30:33.486 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:30:33.486 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:30:33.486 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:33.486 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.483+0000 7fa0ff3f6640 1 librados: starting msgr at 2024-03-25T18:30:33.486 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.483+0000 7fa0ff3f6640 1 librados: starting objecter 2024-03-25T18:30:33.486 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.484+0000 7fa0ff3f6640 1 Processor -- start 2024-03-25T18:30:33.486 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.484+0000 7fa0ff3f6640 1 -- start start 2024-03-25T18:30:33.486 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.484+0000 7fa0ff3f6640 1 librados: setting wanted keys 2024-03-25T18:30:33.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.484+0000 7fa0ff3f6640 1 librados: calling monclient init 2024-03-25T18:30:33.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.484+0000 7fa0ff3f6640 10 monclient: init 2024-03-25T18:30:33.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.484+0000 7fa0ff3f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:33.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.484+0000 7fa0ff3f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:30:33.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.484+0000 7fa0ff3f6640 10 monclient(hunting): picked mon.noname-b con 0x7fa0f80b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:30:33.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.485+0000 7fa0ff3f6640 10 monclient(hunting): picked mon.noname-a con 0x7fa0f80b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:30:33.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.485+0000 7fa0ff3f6640 10 monclient(hunting): picked mon.noname-c con 0x7fa0f813c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:30:33.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.485+0000 7fa0ff3f6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f80af4f0 con 0x7fa0f80b4ff0 2024-03-25T18:30:33.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.485+0000 7fa0ff3f6640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f80ada40 con 0x7fa0f80b4670 2024-03-25T18:30:33.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.485+0000 7fa0ff3f6640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f80afcd0 con 0x7fa0f813c480 2024-03-25T18:30:33.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.485+0000 7fa0ff3f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:33.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.485+0000 7fa0ff3f6640 10 monclient(hunting): authenticate will time out at 7687.745541s 2024-03-25T18:30:33.497 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.495+0000 7fa0fd993640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fa0f813c480 0x7fa0f80b10a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:41212/0 (socket says 172.21.15.3:41212) 2024-03-25T18:30:33.497 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.496+0000 7fa0fd993640 1 -- 172.21.15.3:0/2877505898 learned_addr learned my addr 172.21.15.3:0/2877505898 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:30:33.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.497+0000 7fa0edffb640 10 client.?.objecter ms_handle_connect 0x7fa0f813c480 2024-03-25T18:30:33.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.497+0000 7fa0edffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:33.500 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.499+0000 7fa0edffb640 10 client.?.objecter ms_handle_connect 0x7fa0f80b4670 2024-03-25T18:30:33.500 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.499+0000 7fa0edffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:33.514 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.513+0000 7fa0edffb640 10 client.?.objecter ms_handle_connect 0x7fa0f80b4ff0 2024-03-25T18:30:33.514 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.513+0000 7fa0edffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:33.520 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.518+0000 7fa0edffb640 1 -- 172.21.15.3:0/2877505898 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 459493338 0 0) 0x7fa0f80afcd0 con 0x7fa0f813c480 2024-03-25T18:30:33.520 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.519+0000 7fa0edffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:33.520 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.519+0000 7fa0edffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:33.520 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.519+0000 7fa0edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:33.520 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.519+0000 7fa0edffb640 10 monclient(hunting): my global_id is 4445 2024-03-25T18:30:33.520 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.519+0000 7fa0edffb640 1 -- 172.21.15.3:0/2877505898 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0dc004eb0 con 0x7fa0f813c480 2024-03-25T18:30:33.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.521+0000 7fa0edffb640 1 -- 172.21.15.3:0/2877505898 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873220211 0 0) 0x7fa0f80af4f0 con 0x7fa0f80b4ff0 2024-03-25T18:30:33.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.521+0000 7fa0edffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:33.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.521+0000 7fa0edffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:33.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.521+0000 7fa0edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:33.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.521+0000 7fa0edffb640 10 monclient(hunting): my global_id is 4482 2024-03-25T18:30:33.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.521+0000 7fa0edffb640 1 -- 172.21.15.3:0/2877505898 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0f80afcd0 con 0x7fa0f80b4ff0 2024-03-25T18:30:33.524 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.522+0000 7fa0edffb640 1 -- 172.21.15.3:0/2877505898 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4037125026 0 0) 0x7fa0f80ada40 con 0x7fa0f80b4670 2024-03-25T18:30:33.524 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.522+0000 7fa0edffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:33.524 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.522+0000 7fa0edffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:33.524 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.522+0000 7fa0edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:33.524 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.522+0000 7fa0edffb640 10 monclient(hunting): my global_id is 4435 2024-03-25T18:30:33.524 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.522+0000 7fa0edffb640 1 -- 172.21.15.3:0/2877505898 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0f80af4f0 con 0x7fa0f80b4670 2024-03-25T18:30:33.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.532+0000 7fa0edffb640 1 -- 172.21.15.3:0/2877505898 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2718814467 0 0) 0x7fa0dc004eb0 con 0x7fa0f813c480 2024-03-25T18:30:33.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.532+0000 7fa0edffb640 1 -- 172.21.15.3:0/2877505898 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0f80ada40 con 0x7fa0f813c480 2024-03-25T18:30:33.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.532+0000 7fa0edffb640 1 -- 172.21.15.3:0/2877505898 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fa0f0001c30 con 0x7fa0f813c480 2024-03-25T18:30:33.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.532+0000 7fa0edffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:33.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.532+0000 7fa0edffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:33.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.532+0000 7fa0edffb640 10 monclient(hunting): dump: 2024-03-25T18:30:33.535 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:33.535 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:33.535 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:33.535 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:33.535 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:33.535 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:33.535 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:33.535 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:33.535 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:33.535 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:33.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.532+0000 7fa0edffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:33.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.533+0000 7fa0edffb640 1 -- 172.21.15.3:0/2877505898 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2791206680 0 0) 0x7fa0f80af4f0 con 0x7fa0f80b4670 2024-03-25T18:30:33.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.533+0000 7fa0edffb640 1 -- 172.21.15.3:0/2877505898 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0dc004eb0 con 0x7fa0f80b4670 2024-03-25T18:30:33.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.533+0000 7fa0edffb640 1 -- 172.21.15.3:0/2877505898 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fa0f4001930 con 0x7fa0f80b4670 2024-03-25T18:30:33.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.533+0000 7fa0edffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:33.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.533+0000 7fa0edffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:33.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.533+0000 7fa0edffb640 10 monclient(hunting): dump: 2024-03-25T18:30:33.536 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:33.536 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:33.536 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:33.536 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:33.536 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:33.536 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:33.536 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:33.536 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:33.536 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:33.536 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:33.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.540+0000 7fa0edffb640 1 -- 172.21.15.3:0/2877505898 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 62581035 0 0) 0x7fa0f80afcd0 con 0x7fa0f80b4ff0 2024-03-25T18:30:33.542 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.540+0000 7fa0edffb640 1 -- 172.21.15.3:0/2877505898 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0f80af4f0 con 0x7fa0f80b4ff0 2024-03-25T18:30:33.542 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.540+0000 7fa0edffb640 1 -- 172.21.15.3:0/2877505898 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fa0e8001b40 con 0x7fa0f80b4ff0 2024-03-25T18:30:33.542 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.540+0000 7fa0edffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:33.542 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.540+0000 7fa0edffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:33.542 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.540+0000 7fa0edffb640 10 monclient(hunting): dump: 2024-03-25T18:30:33.542 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:33.542 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:33.542 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:33.542 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:33.542 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:33.542 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:33.542 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:33.542 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:33.543 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:33.543 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:33.552 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.551+0000 7fa0edffb640 1 -- 172.21.15.3:0/2877505898 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3709031181 0 0) 0x7fa0dc004eb0 con 0x7fa0f80b4670 2024-03-25T18:30:33.552 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.551+0000 7fa0edffb640 1 -- 172.21.15.3:0/2877505898 >> v1:172.21.15.3:6790/0 conn(0x7fa0f813c480 legacy=0x7fa0f80b10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:33.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.551+0000 7fa0edffb640 1 -- 172.21.15.3:0/2877505898 >> v1:172.21.15.3:6789/0 conn(0x7fa0f80b4ff0 legacy=0x7fa0f80b0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:33.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.551+0000 7fa0edffb640 10 monclient: _finish_hunting 0 2024-03-25T18:30:33.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.551+0000 7fa0edffb640 1 monclient: found mon.b 2024-03-25T18:30:33.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.551+0000 7fa0edffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:33.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.551+0000 7fa0edffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:30:33.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.551+0000 7fa0edffb640 1 -- 172.21.15.3:0/2877505898 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f80affb0 con 0x7fa0f80b4670 2024-03-25T18:30:33.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.551+0000 7fa0edffb640 10 monclient: _finish_auth 0 2024-03-25T18:30:33.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.551+0000 7fa0edffb640 10 monclient: _check_auth_tickets 2024-03-25T18:30:33.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.551+0000 7fa0edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:33.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.551+0000 7fa0ff3f6640 5 monclient: authenticate success, global_id 4435 2024-03-25T18:30:33.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.551+0000 7fa0ff3f6640 10 monclient: _renew_subs 2024-03-25T18:30:33.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.551+0000 7fa0ff3f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:30:33.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.551+0000 7fa0ff3f6640 1 -- 172.21.15.3:0/2877505898 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f80b0290 con 0x7fa0f80b4670 2024-03-25T18:30:33.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.551+0000 7fa0ff3f6640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:33.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.551+0000 7fa0ff3f6640 10 monclient: _renew_subs 2024-03-25T18:30:33.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.551+0000 7fa0ff3f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:30:33.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.551+0000 7fa0ff3f6640 1 -- 172.21.15.3:0/2877505898 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f80aefd0 con 0x7fa0f80b4670 2024-03-25T18:30:33.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.551+0000 7fa0ff3f6640 1 librados: init done 2024-03-25T18:30:33.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.552+0000 7fa0ff3f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:33.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.552+0000 7fa0ff3f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:33.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.552+0000 7fa0ff3f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:30:33.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.552+0000 7fa0ff3f6640 1 -- 172.21.15.3:0/2877505898 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0f806dbd0 con 0x7fa0f80b4670 2024-03-25T18:30:33.843 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.842+0000 7fa0edffb640 1 -- 172.21.15.3:0/2877505898 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0f4003610 con 0x7fa0f80b4670 2024-03-25T18:30:33.843 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.842+0000 7fa0edffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:33.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.842+0000 7fa0edffb640 1 -- 172.21.15.3:0/2877505898 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fa0f4001d80 con 0x7fa0f80b4670 2024-03-25T18:30:33.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.842+0000 7fa0edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:33.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.842+0000 7fa0edffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:33.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.842+0000 7fa0edffb640 10 monclient: dump: 2024-03-25T18:30:33.844 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:33.844 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:33.844 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:33.844 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:33.844 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:33.844 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:33.844 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:33.844 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:33.844 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:33.844 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:33.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.842+0000 7fa0ef7fe640 4 set_mon_vals no callback set 2024-03-25T18:30:33.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.843+0000 7fa0edffb640 1 -- 172.21.15.3:0/2877505898 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7fa0f4019850 con 0x7fa0f80b4670 2024-03-25T18:30:33.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.843+0000 7fa0fc991640 10 monclient: get_auth_request con 0x7fa0dc06e160 auth_method 0 2024-03-25T18:30:33.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.844+0000 7fa0edffb640 10 client.4435.objecter ms_handle_connect 0x7fa0dc06e160 2024-03-25T18:30:33.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.848+0000 7fa0edffb640 1 -- 172.21.15.3:0/2877505898 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7fa0f4053510 con 0x7fa0f80b4670 2024-03-25T18:30:33.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.848+0000 7fa0edffb640 10 client.4435.objecter ms_dispatch 0x7fa0f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:33.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.848+0000 7fa0edffb640 3 client.4435.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:33.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.848+0000 7fa0edffb640 3 client.4435.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:33.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.848+0000 7fa0edffb640 20 client.4435.objecter dump_active .. 0 homeless 2024-03-25T18:30:33.852 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.851+0000 7fa0edffb640 1 -- 172.21.15.3:0/2877505898 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa0f404ee50 con 0x7fa0f80b4670 2024-03-25T18:30:33.852 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.851+0000 7fa0edffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:33.852 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:33.851+0000 7fa0edffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:34.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:34.099+0000 7fa0ff3f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:34.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:34.099+0000 7fa0ff3f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:34.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:34.099+0000 7fa0ff3f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:30:34.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:34.099+0000 7fa0ff3f6640 1 -- 172.21.15.3:0/2877505898 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0f80688c0 con 0x7fa0f80b4670 2024-03-25T18:30:34.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:34.241+0000 7fa0edffb640 1 -- 172.21.15.3:0/2877505898 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7fa0f4050f50 con 0x7fa0f80b4670 2024-03-25T18:30:34.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:34.241+0000 7fa0edffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:34.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:34.241+0000 7fa0edffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:34.242 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:30:34.243 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:34.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:34.243+0000 7fa0ff3f6640 10 librados: watch_flush enter 2024-03-25T18:30:34.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:34.243+0000 7fa0ff3f6640 10 librados: watch_flush exit 2024-03-25T18:30:34.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:34.243+0000 7fa0ff3f6640 20 client.4435.objecter shutdown clearing up homeless session... 2024-03-25T18:30:34.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:34.243+0000 7fa0ff3f6640 10 client.4435.objecter successfully canceled tick 2024-03-25T18:30:34.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:34.244+0000 7fa0ff3f6640 1 -- 172.21.15.3:0/2877505898 >> v1:172.21.15.3:6800/4195140966 conn(0x7fa0dc06e160 legacy=0x7fa0dc070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:34.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:34.244+0000 7fa0ff3f6640 10 monclient: shutdown 2024-03-25T18:30:34.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:34.244+0000 7fa0ff3f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:34.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:34.244+0000 7fa0ff3f6640 1 -- 172.21.15.3:0/2877505898 >> v1:172.21.15.16:6789/0 conn(0x7fa0f80b4670 legacy=0x7fa0f813a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:34.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:34.244+0000 7fa0ff3f6640 1 -- 172.21.15.3:0/2877505898 shutdown_connections 2024-03-25T18:30:34.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:34.244+0000 7fa0ff3f6640 1 -- 172.21.15.3:0/2877505898 >> 172.21.15.3:0/2877505898 conn(0x7fa0f80617a0 msgr2=0x7fa0f80b71c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:34.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:34.244+0000 7fa0ff3f6640 1 -- 172.21.15.3:0/2877505898 shutdown_connections 2024-03-25T18:30:34.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:34.244+0000 7fa0ff3f6640 1 -- 172.21.15.3:0/2877505898 wait complete. 2024-03-25T18:30:34.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:34.244+0000 7fa0ff3f6640 1 librados: shutdown 2024-03-25T18:30:34.256 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:41.063 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:41.175 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.174+0000 7ffa283d6640 10 monclient: get_monmap_and_config 2024-03-25T18:30:41.175 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.174+0000 7ffa283d6640 10 monclient: build_initial_monmap 2024-03-25T18:30:41.175 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.174+0000 7ffa283d6640 10 monclient: monmap: 2024-03-25T18:30:41.175 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:30:41.175 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:41.175 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:30:41.175240+0000 2024-03-25T18:30:41.176 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:30:41.175240+0000 2024-03-25T18:30:41.176 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:41.176 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:41.176 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:30:41.176 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:30:41.176 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:30:41.176 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:41.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.174+0000 7ffa283d6640 1 Processor -- start 2024-03-25T18:30:41.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.174+0000 7ffa283d6640 1 -- start start 2024-03-25T18:30:41.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.174+0000 7ffa283d6640 10 monclient: init 2024-03-25T18:30:41.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.175+0000 7ffa283d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:41.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.175+0000 7ffa283d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:30:41.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.175+0000 7ffa283d6640 10 monclient(hunting): picked mon.noname-b con 0x7ffa200b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:30:41.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.175+0000 7ffa283d6640 10 monclient(hunting): picked mon.noname-a con 0x7ffa200b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:30:41.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.175+0000 7ffa283d6640 10 monclient(hunting): picked mon.noname-c con 0x7ffa2013c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:30:41.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.175+0000 7ffa283d6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa200ab0e0 con 0x7ffa200b4670 2024-03-25T18:30:41.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.175+0000 7ffa283d6640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa200ab3c0 con 0x7ffa2013c480 2024-03-25T18:30:41.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.175+0000 7ffa283d6640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa200a9f30 con 0x7ffa200b4ff0 2024-03-25T18:30:41.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.175+0000 7ffa283d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:41.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.175+0000 7ffa283d6640 10 monclient(hunting): authenticate will time out at 7695.436304s 2024-03-25T18:30:41.182 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.181+0000 7ffa25971640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ffa200b4670 0x7ffa201425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:40746/0 (socket says 172.21.15.3:40746) 2024-03-25T18:30:41.183 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.182+0000 7ffa25971640 1 -- 172.21.15.3:0/2299273801 learned_addr learned my addr 172.21.15.3:0/2299273801 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:30:41.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.192+0000 7ffa25170640 1 -- 172.21.15.3:0/2299273801 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1841595454 0 0) 0x7ffa200ab0e0 con 0x7ffa200b4670 2024-03-25T18:30:41.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.192+0000 7ffa25170640 10 monclient(hunting): _negotiate 2024-03-25T18:30:41.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.192+0000 7ffa25170640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:41.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.192+0000 7ffa25170640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:41.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.192+0000 7ffa25170640 10 monclient(hunting): my global_id is 4485 2024-03-25T18:30:41.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.192+0000 7ffa25170640 1 -- 172.21.15.3:0/2299273801 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa0c003520 con 0x7ffa200b4670 2024-03-25T18:30:41.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.203+0000 7ffa25170640 1 -- 172.21.15.3:0/2299273801 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2917013178 0 0) 0x7ffa200a9f30 con 0x7ffa200b4ff0 2024-03-25T18:30:41.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.203+0000 7ffa25170640 10 monclient(hunting): _negotiate 2024-03-25T18:30:41.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.203+0000 7ffa25170640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:41.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.203+0000 7ffa25170640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:41.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.203+0000 7ffa25170640 10 monclient(hunting): my global_id is 4448 2024-03-25T18:30:41.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.203+0000 7ffa25170640 1 -- 172.21.15.3:0/2299273801 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa200ab0e0 con 0x7ffa200b4ff0 2024-03-25T18:30:41.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.207+0000 7ffa25170640 1 -- 172.21.15.3:0/2299273801 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2683810592 0 0) 0x7ffa200ab3c0 con 0x7ffa2013c480 2024-03-25T18:30:41.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.207+0000 7ffa25170640 10 monclient(hunting): _negotiate 2024-03-25T18:30:41.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.207+0000 7ffa25170640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:41.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.207+0000 7ffa25170640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:41.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.207+0000 7ffa25170640 10 monclient(hunting): my global_id is 4438 2024-03-25T18:30:41.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.207+0000 7ffa25170640 1 -- 172.21.15.3:0/2299273801 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa200a9f30 con 0x7ffa2013c480 2024-03-25T18:30:41.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.207+0000 7ffa25170640 1 -- 172.21.15.3:0/2299273801 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1135401702 0 0) 0x7ffa0c003520 con 0x7ffa200b4670 2024-03-25T18:30:41.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.207+0000 7ffa25170640 1 -- 172.21.15.3:0/2299273801 >> v1:172.21.15.3:6790/0 conn(0x7ffa200b4ff0 legacy=0x7ffa2013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:41.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.207+0000 7ffa25170640 1 -- 172.21.15.3:0/2299273801 >> v1:172.21.15.16:6789/0 conn(0x7ffa2013c480 legacy=0x7ffa2014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:41.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.207+0000 7ffa25170640 10 monclient: _finish_hunting 0 2024-03-25T18:30:41.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.207+0000 7ffa25170640 1 monclient: found mon.noname-a 2024-03-25T18:30:41.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.207+0000 7ffa25170640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:41.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.207+0000 7ffa25170640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:30:41.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.207+0000 7ffa25170640 1 -- 172.21.15.3:0/2299273801 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa200a9b70 con 0x7ffa200b4670 2024-03-25T18:30:41.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.207+0000 7ffa25170640 10 monclient: _finish_auth 0 2024-03-25T18:30:41.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.207+0000 7ffa25170640 10 monclient: _check_auth_tickets 2024-03-25T18:30:41.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.207+0000 7ffa25170640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:41.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.207+0000 7ffa25170640 1 -- 172.21.15.3:0/2299273801 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7ffa10001690 con 0x7ffa200b4670 2024-03-25T18:30:41.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.207+0000 7ffa25170640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:41.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.207+0000 7ffa25170640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:41.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.207+0000 7ffa25170640 10 monclient: dump: 2024-03-25T18:30:41.209 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:41.210 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:41.210 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:41.210 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:41.210 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:41.210 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:41.210 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:41.210 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:41.210 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:41.210 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:41.210 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.207+0000 7ffa283d6640 5 monclient: authenticate success, global_id 4485 2024-03-25T18:30:41.210 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.207+0000 7ffa283d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:41.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.213+0000 7ffa25170640 1 -- 172.21.15.3:0/2299273801 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa100017f0 con 0x7ffa200b4670 2024-03-25T18:30:41.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.213+0000 7ffa25170640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:41.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.213+0000 7ffa25170640 1 -- 172.21.15.3:0/2299273801 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7ffa10001c00 con 0x7ffa200b4670 2024-03-25T18:30:41.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.213+0000 7ffa25170640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:41.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.213+0000 7ffa25170640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:41.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.213+0000 7ffa25170640 10 monclient: dump: 2024-03-25T18:30:41.214 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:41.214 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:41.214 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:41.214 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:41.214 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:41.214 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:41.214 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:41.214 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:41.214 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:41.214 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:41.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.213+0000 7ffa283d6640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:41.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.213+0000 7ffa283d6640 4 set_mon_vals no callback set 2024-03-25T18:30:41.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.213+0000 7ffa283d6640 10 monclient: shutdown 2024-03-25T18:30:41.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.213+0000 7ffa283d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:41.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.213+0000 7ffa283d6640 1 -- 172.21.15.3:0/2299273801 >> v1:172.21.15.3:6789/0 conn(0x7ffa200b4670 legacy=0x7ffa201425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:41.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.213+0000 7ffa283d6640 1 -- 172.21.15.3:0/2299273801 shutdown_connections 2024-03-25T18:30:41.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.213+0000 7ffa283d6640 1 -- 172.21.15.3:0/2299273801 >> 172.21.15.3:0/2299273801 conn(0x7ffa200617a0 msgr2=0x7ffa20061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:41.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.213+0000 7ffa283d6640 1 -- 172.21.15.3:0/2299273801 shutdown_connections 2024-03-25T18:30:41.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.213+0000 7ffa283d6640 1 -- 172.21.15.3:0/2299273801 wait complete. 2024-03-25T18:30:41.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.214+0000 7ffa283d6640 10 monclient: build_initial_monmap 2024-03-25T18:30:41.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.214+0000 7ffa283d6640 10 monclient: monmap: 2024-03-25T18:30:41.217 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:30:41.217 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:41.217 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:30:41.217 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:30:41.217 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:41.217 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:30:41.217 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:30:41.217 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:30:41.217 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:30:41.217 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:41.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.214+0000 7ffa283d6640 1 librados: starting msgr at 2024-03-25T18:30:41.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.214+0000 7ffa283d6640 1 librados: starting objecter 2024-03-25T18:30:41.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.214+0000 7ffa283d6640 1 Processor -- start 2024-03-25T18:30:41.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.215+0000 7ffa283d6640 1 -- start start 2024-03-25T18:30:41.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.215+0000 7ffa283d6640 1 librados: setting wanted keys 2024-03-25T18:30:41.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.215+0000 7ffa283d6640 1 librados: calling monclient init 2024-03-25T18:30:41.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.215+0000 7ffa283d6640 10 monclient: init 2024-03-25T18:30:41.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.215+0000 7ffa283d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:41.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.215+0000 7ffa283d6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:30:41.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.215+0000 7ffa283d6640 10 monclient(hunting): picked mon.noname-b con 0x7ffa200b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:30:41.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.215+0000 7ffa283d6640 10 monclient(hunting): picked mon.noname-c con 0x7ffa200b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:30:41.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.215+0000 7ffa283d6640 10 monclient(hunting): picked mon.noname-a con 0x7ffa2013c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:30:41.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.215+0000 7ffa283d6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa20144740 con 0x7ffa2013c480 2024-03-25T18:30:41.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.215+0000 7ffa283d6640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa20130b60 con 0x7ffa200b4670 2024-03-25T18:30:41.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.215+0000 7ffa283d6640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa2013c930 con 0x7ffa200b4ff0 2024-03-25T18:30:41.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.215+0000 7ffa283d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:41.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.215+0000 7ffa283d6640 10 monclient(hunting): authenticate will time out at 7695.476182s 2024-03-25T18:30:41.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.227+0000 7ffa26973640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ffa2013c480 0x7ffa20130450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:40764/0 (socket says 172.21.15.3:40764) 2024-03-25T18:30:41.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.227+0000 7ffa26973640 1 -- 172.21.15.3:0/1948938417 learned_addr learned my addr 172.21.15.3:0/1948938417 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:30:41.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.229+0000 7ffa16ffd640 10 client.?.objecter ms_handle_connect 0x7ffa2013c480 2024-03-25T18:30:41.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.229+0000 7ffa16ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:41.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.231+0000 7ffa16ffd640 10 client.?.objecter ms_handle_connect 0x7ffa200b4ff0 2024-03-25T18:30:41.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.231+0000 7ffa16ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:41.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.233+0000 7ffa16ffd640 10 client.?.objecter ms_handle_connect 0x7ffa200b4670 2024-03-25T18:30:41.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.233+0000 7ffa16ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:41.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.248+0000 7ffa16ffd640 1 -- 172.21.15.3:0/1948938417 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4066201917 0 0) 0x7ffa20144740 con 0x7ffa2013c480 2024-03-25T18:30:41.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.248+0000 7ffa16ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:41.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.248+0000 7ffa16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:41.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.248+0000 7ffa16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:41.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.248+0000 7ffa16ffd640 10 monclient(hunting): my global_id is 4488 2024-03-25T18:30:41.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.248+0000 7ffa16ffd640 1 -- 172.21.15.3:0/1948938417 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa04004eb0 con 0x7ffa2013c480 2024-03-25T18:30:41.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.262+0000 7ffa16ffd640 1 -- 172.21.15.3:0/1948938417 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1469338199 0 0) 0x7ffa20130b60 con 0x7ffa200b4670 2024-03-25T18:30:41.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.262+0000 7ffa16ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:41.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.262+0000 7ffa16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:41.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.262+0000 7ffa16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:41.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.262+0000 7ffa16ffd640 10 monclient(hunting): my global_id is 4441 2024-03-25T18:30:41.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.262+0000 7ffa26973640 1 -- 172.21.15.3:0/1948938417 >> v1:172.21.15.3:6789/0 conn(0x7ffa2013c480 legacy=0x7ffa20130450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T18:30:41.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.262+0000 7ffa16ffd640 1 -- 172.21.15.3:0/1948938417 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa20144740 con 0x7ffa200b4670 2024-03-25T18:30:41.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.262+0000 7ffa26973640 1 -- 172.21.15.3:0/1948938417 >> v1:172.21.15.3:6789/0 conn(0x7ffa2013c480 legacy=0x7ffa20130450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:30:41.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.262+0000 7ffa26973640 1 --1- 172.21.15.3:0/1948938417 >> v1:172.21.15.3:6789/0 conn(0x7ffa2013c480 0x7ffa20130450 :-1 s=OPENED pgs=37725 cs=1 l=1).handle_message read tag failed 2024-03-25T18:30:41.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.262+0000 7ffa26973640 1 --1- 172.21.15.3:0/1948938417 >> v1:172.21.15.3:6789/0 conn(0x7ffa2013c480 0x7ffa20130450 :-1 s=OPENED pgs=37725 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:30:41.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.262+0000 7ffa16ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.3:6789/0 2024-03-25T18:30:41.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.264+0000 7ffa16ffd640 1 -- 172.21.15.3:0/1948938417 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2938794395 0 0) 0x7ffa2013c930 con 0x7ffa200b4ff0 2024-03-25T18:30:41.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.264+0000 7ffa16ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:41.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.264+0000 7ffa16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:41.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.264+0000 7ffa16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:41.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.264+0000 7ffa16ffd640 10 monclient(hunting): my global_id is 4451 2024-03-25T18:30:41.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.264+0000 7ffa16ffd640 1 -- 172.21.15.3:0/1948938417 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa20130b60 con 0x7ffa200b4ff0 2024-03-25T18:30:41.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.285+0000 7ffa16ffd640 1 -- 172.21.15.3:0/1948938417 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3820122927 0 0) 0x7ffa20130b60 con 0x7ffa200b4ff0 2024-03-25T18:30:41.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.285+0000 7ffa16ffd640 1 -- 172.21.15.3:0/1948938417 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa2013c930 con 0x7ffa200b4ff0 2024-03-25T18:30:41.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.285+0000 7ffa16ffd640 1 -- 172.21.15.3:0/1948938417 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7ffa10001b00 con 0x7ffa200b4ff0 2024-03-25T18:30:41.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.285+0000 7ffa16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:41.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.285+0000 7ffa16ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:41.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.285+0000 7ffa16ffd640 10 monclient(hunting): dump: 2024-03-25T18:30:41.287 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:41.287 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:41.287 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:41.287 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:41.287 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:41.287 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:41.287 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:41.287 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:41.288 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:41.288 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:41.288 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.285+0000 7ffa16ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:41.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.302+0000 7ffa16ffd640 1 -- 172.21.15.3:0/1948938417 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 644071478 0 0) 0x7ffa2013c930 con 0x7ffa200b4ff0 2024-03-25T18:30:41.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.302+0000 7ffa16ffd640 1 -- 172.21.15.3:0/1948938417 >> v1:172.21.15.16:6789/0 conn(0x7ffa200b4670 legacy=0x7ffa20145d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:41.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.302+0000 7ffa16ffd640 1 -- 172.21.15.3:0/1948938417 >> v1:172.21.15.3:6789/0 conn(0x7ffa2013c480 legacy=0x7ffa20130450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:30:41.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.302+0000 7ffa16ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:41.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.302+0000 7ffa16ffd640 1 monclient: found mon.c 2024-03-25T18:30:41.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.302+0000 7ffa16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:41.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.302+0000 7ffa16ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:30:41.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.302+0000 7ffa16ffd640 1 -- 172.21.15.3:0/1948938417 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa20144970 con 0x7ffa200b4ff0 2024-03-25T18:30:41.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.302+0000 7ffa16ffd640 10 monclient: _finish_auth 0 2024-03-25T18:30:41.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.302+0000 7ffa16ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:41.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.302+0000 7ffa16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:41.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.302+0000 7ffa283d6640 5 monclient: authenticate success, global_id 4451 2024-03-25T18:30:41.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.302+0000 7ffa283d6640 10 monclient: _renew_subs 2024-03-25T18:30:41.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.302+0000 7ffa283d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:30:41.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.302+0000 7ffa283d6640 1 -- 172.21.15.3:0/1948938417 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa20144c50 con 0x7ffa200b4ff0 2024-03-25T18:30:41.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.302+0000 7ffa283d6640 10 client.4451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:41.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.302+0000 7ffa283d6640 10 monclient: _renew_subs 2024-03-25T18:30:41.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.302+0000 7ffa283d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:30:41.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.302+0000 7ffa283d6640 1 -- 172.21.15.3:0/1948938417 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa2013f280 con 0x7ffa200b4ff0 2024-03-25T18:30:41.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.302+0000 7ffa283d6640 1 librados: init done 2024-03-25T18:30:41.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.304+0000 7ffa283d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:41.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.304+0000 7ffa283d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:41.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.304+0000 7ffa283d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:30:41.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.304+0000 7ffa283d6640 1 -- 172.21.15.3:0/1948938417 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa2006dbd0 con 0x7ffa200b4ff0 2024-03-25T18:30:41.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.306+0000 7ffa16ffd640 1 -- 172.21.15.3:0/1948938417 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa10001c60 con 0x7ffa200b4ff0 2024-03-25T18:30:41.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.306+0000 7ffa16ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:41.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.306+0000 7ffa16ffd640 1 -- 172.21.15.3:0/1948938417 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7ffa10004c30 con 0x7ffa200b4ff0 2024-03-25T18:30:41.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.306+0000 7ffa16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:41.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.306+0000 7ffa16ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:41.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.306+0000 7ffa16ffd640 10 monclient: dump: 2024-03-25T18:30:41.308 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:41.308 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:41.308 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:41.308 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:41.308 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:41.308 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:41.308 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:41.308 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:41.309 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:41.309 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:41.309 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.306+0000 7ffa2496f640 4 set_mon_vals no callback set 2024-03-25T18:30:41.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.324+0000 7ffa16ffd640 1 -- 172.21.15.3:0/1948938417 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7ffa1001ac00 con 0x7ffa200b4ff0 2024-03-25T18:30:41.326 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.325+0000 7ffa26172640 10 monclient: get_auth_request con 0x7ffa0406ddb0 auth_method 0 2024-03-25T18:30:41.326 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.325+0000 7ffa16ffd640 10 client.4451.objecter ms_handle_connect 0x7ffa0406ddb0 2024-03-25T18:30:41.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.331+0000 7ffa16ffd640 1 -- 172.21.15.3:0/1948938417 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7ffa10054750 con 0x7ffa200b4ff0 2024-03-25T18:30:41.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.331+0000 7ffa16ffd640 10 client.4451.objecter ms_dispatch 0x7ffa20000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:41.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.331+0000 7ffa16ffd640 3 client.4451.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:41.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.331+0000 7ffa16ffd640 3 client.4451.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:41.332 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.331+0000 7ffa16ffd640 20 client.4451.objecter dump_active .. 0 homeless 2024-03-25T18:30:41.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.615+0000 7ffa16ffd640 1 -- 172.21.15.3:0/1948938417 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffa10050260 con 0x7ffa200b4ff0 2024-03-25T18:30:41.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.615+0000 7ffa16ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:41.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.615+0000 7ffa16ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:41.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.888+0000 7ffa283d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:41.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.888+0000 7ffa283d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:41.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.888+0000 7ffa283d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:30:41.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.888+0000 7ffa283d6640 1 -- 172.21.15.3:0/1948938417 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa200688c0 con 0x7ffa200b4ff0 2024-03-25T18:30:41.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.936+0000 7ffa16ffd640 1 -- 172.21.15.3:0/1948938417 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7ffa10052190 con 0x7ffa200b4ff0 2024-03-25T18:30:41.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.936+0000 7ffa16ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:41.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.936+0000 7ffa16ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:41.937 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:30:41.937 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:41.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.938+0000 7ffa283d6640 10 librados: watch_flush enter 2024-03-25T18:30:41.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.938+0000 7ffa283d6640 10 librados: watch_flush exit 2024-03-25T18:30:41.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.938+0000 7ffa283d6640 20 client.4451.objecter shutdown clearing up homeless session... 2024-03-25T18:30:41.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.938+0000 7ffa283d6640 10 client.4451.objecter successfully canceled tick 2024-03-25T18:30:41.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.938+0000 7ffa283d6640 1 -- 172.21.15.3:0/1948938417 >> v1:172.21.15.3:6800/4195140966 conn(0x7ffa0406ddb0 legacy=0x7ffa04070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:41.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.938+0000 7ffa283d6640 10 monclient: shutdown 2024-03-25T18:30:41.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.938+0000 7ffa283d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:41.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.938+0000 7ffa283d6640 1 -- 172.21.15.3:0/1948938417 >> v1:172.21.15.3:6790/0 conn(0x7ffa200b4ff0 legacy=0x7ffa2012fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:41.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.938+0000 7ffa283d6640 1 -- 172.21.15.3:0/1948938417 shutdown_connections 2024-03-25T18:30:41.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.938+0000 7ffa283d6640 1 -- 172.21.15.3:0/1948938417 >> 172.21.15.3:0/1948938417 conn(0x7ffa200617a0 msgr2=0x7ffa201407f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:41.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.938+0000 7ffa283d6640 1 -- 172.21.15.3:0/1948938417 shutdown_connections 2024-03-25T18:30:41.940 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.938+0000 7ffa283d6640 1 -- 172.21.15.3:0/1948938417 wait complete. 2024-03-25T18:30:41.940 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:41.938+0000 7ffa283d6640 1 librados: shutdown 2024-03-25T18:30:41.950 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:41.993 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:05:55.721 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:42.045 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:05:55.774 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:42.045 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:05:55.774 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:42.081 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:05:55.810 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:42.098 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:05:55.790 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:42.105 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:05:55.835 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:42.106 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:05:55.799 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:42.134 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:05:55.826 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:42.142 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:05:55.871 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:48.757 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:48.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.875+0000 7fd9684e6640 10 monclient: get_monmap_and_config 2024-03-25T18:30:48.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.875+0000 7fd9684e6640 10 monclient: build_initial_monmap 2024-03-25T18:30:48.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.875+0000 7fd9684e6640 10 monclient: monmap: 2024-03-25T18:30:48.877 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:30:48.877 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:48.877 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:30:48.876505+0000 2024-03-25T18:30:48.877 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:30:48.876505+0000 2024-03-25T18:30:48.877 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:48.877 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:48.877 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:30:48.877 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:30:48.877 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:30:48.877 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:48.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.876+0000 7fd9684e6640 1 Processor -- start 2024-03-25T18:30:48.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.876+0000 7fd9684e6640 1 -- start start 2024-03-25T18:30:48.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.876+0000 7fd9684e6640 10 monclient: init 2024-03-25T18:30:48.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.877+0000 7fd9684e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:48.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.877+0000 7fd9684e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:30:48.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.877+0000 7fd9684e6640 10 monclient(hunting): picked mon.noname-b con 0x7fd9600b1ed0 addr v1:172.21.15.3:6790/0 2024-03-25T18:30:48.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.877+0000 7fd9684e6640 10 monclient(hunting): picked mon.noname-a con 0x7fd960071db0 addr v1:172.21.15.3:6789/0 2024-03-25T18:30:48.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.877+0000 7fd9684e6640 10 monclient(hunting): picked mon.noname-c con 0x7fd960143f90 addr v1:172.21.15.16:6789/0 2024-03-25T18:30:48.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.877+0000 7fd9684e6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd96006eef0 con 0x7fd960071db0 2024-03-25T18:30:48.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.877+0000 7fd9684e6640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd96006f1d0 con 0x7fd960143f90 2024-03-25T18:30:48.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.877+0000 7fd9684e6640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd96006dfe0 con 0x7fd9600b1ed0 2024-03-25T18:30:48.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.877+0000 7fd9684e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:48.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.877+0000 7fd9684e6640 10 monclient(hunting): authenticate will time out at 7703.137692s 2024-03-25T18:30:48.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.886+0000 7fd966a83640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd960143f90 0x7fd960152000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:42188/0 (socket says 172.21.15.3:42188) 2024-03-25T18:30:48.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.886+0000 7fd966a83640 1 -- 172.21.15.3:0/2014414163 learned_addr learned my addr 172.21.15.3:0/2014414163 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:30:48.901 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.900+0000 7fd965280640 1 -- 172.21.15.3:0/2014414163 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660281190 0 0) 0x7fd96006f1d0 con 0x7fd960143f90 2024-03-25T18:30:48.901 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.900+0000 7fd965280640 10 monclient(hunting): _negotiate 2024-03-25T18:30:48.901 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.900+0000 7fd965280640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:48.901 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.900+0000 7fd965280640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:48.902 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.900+0000 7fd965280640 10 monclient(hunting): my global_id is 4444 2024-03-25T18:30:48.902 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.900+0000 7fd965280640 1 -- 172.21.15.3:0/2014414163 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd94c003520 con 0x7fd960143f90 2024-03-25T18:30:48.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.906+0000 7fd965280640 1 -- 172.21.15.3:0/2014414163 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4216820105 0 0) 0x7fd96006eef0 con 0x7fd960071db0 2024-03-25T18:30:48.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.906+0000 7fd965280640 10 monclient(hunting): _negotiate 2024-03-25T18:30:48.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.906+0000 7fd965280640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:48.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.906+0000 7fd965280640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:48.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.906+0000 7fd965280640 10 monclient(hunting): my global_id is 4491 2024-03-25T18:30:48.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.906+0000 7fd965280640 1 -- 172.21.15.3:0/2014414163 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd96006f1d0 con 0x7fd960071db0 2024-03-25T18:30:48.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.913+0000 7fd965280640 1 -- 172.21.15.3:0/2014414163 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 540208116 0 0) 0x7fd94c003520 con 0x7fd960143f90 2024-03-25T18:30:48.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.913+0000 7fd965280640 1 -- 172.21.15.3:0/2014414163 >> v1:172.21.15.3:6790/0 conn(0x7fd9600b1ed0 legacy=0x7fd960143880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:48.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.913+0000 7fd965280640 1 -- 172.21.15.3:0/2014414163 >> v1:172.21.15.3:6789/0 conn(0x7fd960071db0 legacy=0x7fd960146740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:48.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.913+0000 7fd965280640 10 monclient: _finish_hunting 0 2024-03-25T18:30:48.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.913+0000 7fd965280640 1 monclient: found mon.noname-c 2024-03-25T18:30:48.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.913+0000 7fd965280640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:48.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.913+0000 7fd965280640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:30:48.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.913+0000 7fd965280640 1 -- 172.21.15.3:0/2014414163 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96006e170 con 0x7fd960143f90 2024-03-25T18:30:48.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.913+0000 7fd965280640 10 monclient: _finish_auth 0 2024-03-25T18:30:48.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.913+0000 7fd965280640 10 monclient: _check_auth_tickets 2024-03-25T18:30:48.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.913+0000 7fd965280640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:48.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.913+0000 7fd965280640 1 -- 172.21.15.3:0/2014414163 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd958001a10 con 0x7fd960143f90 2024-03-25T18:30:48.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.913+0000 7fd965280640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:48.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.913+0000 7fd965280640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:48.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.913+0000 7fd965280640 10 monclient: dump: 2024-03-25T18:30:48.916 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:48.916 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:48.916 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:48.916 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:48.916 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:48.916 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:48.916 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:48.916 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:48.916 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:48.916 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:48.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.913+0000 7fd9684e6640 5 monclient: authenticate success, global_id 4444 2024-03-25T18:30:48.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.913+0000 7fd9684e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:48.925 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.924+0000 7fd965280640 1 -- 172.21.15.3:0/2014414163 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd958001b70 con 0x7fd960143f90 2024-03-25T18:30:48.925 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.924+0000 7fd965280640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:48.926 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.924+0000 7fd965280640 1 -- 172.21.15.3:0/2014414163 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd958001f00 con 0x7fd960143f90 2024-03-25T18:30:48.926 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.924+0000 7fd965280640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:48.926 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.924+0000 7fd965280640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:48.926 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.924+0000 7fd965280640 10 monclient: dump: 2024-03-25T18:30:48.926 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:48.926 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:48.926 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:48.926 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:48.926 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:48.926 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:48.926 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:48.926 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:48.926 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:48.927 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:48.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.924+0000 7fd9684e6640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:48.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.924+0000 7fd9684e6640 4 set_mon_vals no callback set 2024-03-25T18:30:48.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.925+0000 7fd9684e6640 10 monclient: shutdown 2024-03-25T18:30:48.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.925+0000 7fd9684e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:48.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.925+0000 7fd9684e6640 1 -- 172.21.15.3:0/2014414163 >> v1:172.21.15.16:6789/0 conn(0x7fd960143f90 legacy=0x7fd960152000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:48.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.925+0000 7fd9684e6640 1 -- 172.21.15.3:0/2014414163 shutdown_connections 2024-03-25T18:30:48.929 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.925+0000 7fd9684e6640 1 -- 172.21.15.3:0/2014414163 >> 172.21.15.3:0/2014414163 conn(0x7fd9600617a0 msgr2=0x7fd960061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:48.929 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.925+0000 7fd9684e6640 1 -- 172.21.15.3:0/2014414163 shutdown_connections 2024-03-25T18:30:48.929 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.925+0000 7fd9684e6640 1 -- 172.21.15.3:0/2014414163 wait complete. 2024-03-25T18:30:48.929 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.926+0000 7fd9684e6640 10 monclient: build_initial_monmap 2024-03-25T18:30:48.929 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.926+0000 7fd9684e6640 10 monclient: monmap: 2024-03-25T18:30:48.929 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:30:48.929 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:48.929 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:30:48.929 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:30:48.929 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:48.929 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:30:48.929 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:30:48.929 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:30:48.929 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:30:48.930 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:48.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.926+0000 7fd9684e6640 1 librados: starting msgr at 2024-03-25T18:30:48.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.926+0000 7fd9684e6640 1 librados: starting objecter 2024-03-25T18:30:48.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.927+0000 7fd9684e6640 1 Processor -- start 2024-03-25T18:30:48.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.927+0000 7fd9684e6640 1 -- start start 2024-03-25T18:30:48.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.927+0000 7fd9684e6640 1 librados: setting wanted keys 2024-03-25T18:30:48.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.927+0000 7fd9684e6640 1 librados: calling monclient init 2024-03-25T18:30:48.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.927+0000 7fd9684e6640 10 monclient: init 2024-03-25T18:30:48.931 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.927+0000 7fd9684e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:48.931 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.928+0000 7fd9684e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:30:48.931 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.928+0000 7fd9684e6640 10 monclient(hunting): picked mon.noname-b con 0x7fd960071db0 addr v1:172.21.15.16:6789/0 2024-03-25T18:30:48.931 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.928+0000 7fd9684e6640 10 monclient(hunting): picked mon.noname-a con 0x7fd9600b1ed0 addr v1:172.21.15.3:6789/0 2024-03-25T18:30:48.931 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.928+0000 7fd9684e6640 10 monclient(hunting): picked mon.noname-c con 0x7fd960143f90 addr v1:172.21.15.3:6790/0 2024-03-25T18:30:48.931 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.928+0000 7fd9684e6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd960147a60 con 0x7fd9600b1ed0 2024-03-25T18:30:48.931 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.928+0000 7fd9684e6640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd960149490 con 0x7fd960071db0 2024-03-25T18:30:48.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.928+0000 7fd9684e6640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9601dc7c0 con 0x7fd960143f90 2024-03-25T18:30:48.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.928+0000 7fd9684e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:48.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.928+0000 7fd9684e6640 10 monclient(hunting): authenticate will time out at 7703.188442s 2024-03-25T18:30:48.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.935+0000 7fd966a83640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fd960143f90 0x7fd9600b8200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:58980/0 (socket says 172.21.15.3:58980) 2024-03-25T18:30:48.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.935+0000 7fd966a83640 1 -- 172.21.15.3:0/2864256460 learned_addr learned my addr 172.21.15.3:0/2864256460 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:30:48.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.936+0000 7fd956ffd640 10 client.?.objecter ms_handle_connect 0x7fd960143f90 2024-03-25T18:30:48.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.936+0000 7fd956ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:48.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.938+0000 7fd956ffd640 10 client.?.objecter ms_handle_connect 0x7fd9600b1ed0 2024-03-25T18:30:48.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.938+0000 7fd956ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:48.941 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.940+0000 7fd956ffd640 1 -- 172.21.15.3:0/2864256460 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2984662861 0 0) 0x7fd9601dc7c0 con 0x7fd960143f90 2024-03-25T18:30:48.941 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.940+0000 7fd956ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:48.941 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.940+0000 7fd956ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:48.941 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.940+0000 7fd956ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:48.942 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.940+0000 7fd956ffd640 10 monclient(hunting): my global_id is 4457 2024-03-25T18:30:48.942 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.940+0000 7fd956ffd640 1 -- 172.21.15.3:0/2864256460 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd944004eb0 con 0x7fd960143f90 2024-03-25T18:30:48.947 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.946+0000 7fd956ffd640 1 -- 172.21.15.3:0/2864256460 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4026463673 0 0) 0x7fd960147a60 con 0x7fd9600b1ed0 2024-03-25T18:30:48.947 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.946+0000 7fd956ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:48.947 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.946+0000 7fd956ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:48.947 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.946+0000 7fd956ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:48.947 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.946+0000 7fd956ffd640 10 monclient(hunting): my global_id is 4494 2024-03-25T18:30:48.947 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.946+0000 7fd956ffd640 1 -- 172.21.15.3:0/2864256460 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9601dc7c0 con 0x7fd9600b1ed0 2024-03-25T18:30:48.950 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.949+0000 7fd956ffd640 1 -- 172.21.15.3:0/2864256460 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1801013195 0 0) 0x7fd944004eb0 con 0x7fd960143f90 2024-03-25T18:30:48.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.949+0000 7fd956ffd640 1 -- 172.21.15.3:0/2864256460 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd960147a60 con 0x7fd960143f90 2024-03-25T18:30:48.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.949+0000 7fd956ffd640 1 -- 172.21.15.3:0/2864256460 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd958001f40 con 0x7fd960143f90 2024-03-25T18:30:48.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.949+0000 7fd956ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:48.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.949+0000 7fd956ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:48.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.949+0000 7fd956ffd640 10 monclient(hunting): dump: 2024-03-25T18:30:48.951 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:48.952 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:48.952 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:48.952 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:48.952 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:48.952 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:48.952 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:48.952 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:48.953 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:48.953 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:48.953 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.949+0000 7fd956ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:48.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.958+0000 7fd956ffd640 10 client.?.objecter ms_handle_connect 0x7fd960071db0 2024-03-25T18:30:48.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.958+0000 7fd956ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:48.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.963+0000 7fd956ffd640 1 -- 172.21.15.3:0/2864256460 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2665081013 0 0) 0x7fd9601dc7c0 con 0x7fd9600b1ed0 2024-03-25T18:30:48.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.963+0000 7fd956ffd640 1 -- 172.21.15.3:0/2864256460 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd944004eb0 con 0x7fd9600b1ed0 2024-03-25T18:30:48.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.963+0000 7fd956ffd640 1 -- 172.21.15.3:0/2864256460 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd950001930 con 0x7fd9600b1ed0 2024-03-25T18:30:48.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.963+0000 7fd956ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:48.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.963+0000 7fd956ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:48.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.963+0000 7fd956ffd640 10 monclient(hunting): dump: 2024-03-25T18:30:48.964 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:48.964 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:48.965 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:48.965 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:48.965 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:48.965 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:48.965 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:48.965 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:48.965 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:48.965 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:48.976 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.975+0000 7fd956ffd640 1 -- 172.21.15.3:0/2864256460 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2340427303 0 0) 0x7fd960147a60 con 0x7fd960143f90 2024-03-25T18:30:48.976 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.975+0000 7fd956ffd640 1 -- 172.21.15.3:0/2864256460 >> v1:172.21.15.16:6789/0 conn(0x7fd960071db0 legacy=0x7fd9601dc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:48.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.975+0000 7fd956ffd640 1 -- 172.21.15.3:0/2864256460 >> v1:172.21.15.3:6789/0 conn(0x7fd9600b1ed0 legacy=0x7fd960147f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:48.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.975+0000 7fd956ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:48.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.975+0000 7fd956ffd640 1 monclient: found mon.c 2024-03-25T18:30:48.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.975+0000 7fd956ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:48.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.975+0000 7fd956ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:30:48.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.975+0000 7fd956ffd640 1 -- 172.21.15.3:0/2864256460 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9601dc950 con 0x7fd960143f90 2024-03-25T18:30:48.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.975+0000 7fd956ffd640 10 monclient: _finish_auth 0 2024-03-25T18:30:48.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.975+0000 7fd956ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:48.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.975+0000 7fd956ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:48.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.975+0000 7fd9684e6640 5 monclient: authenticate success, global_id 4457 2024-03-25T18:30:48.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.975+0000 7fd9684e6640 10 monclient: _renew_subs 2024-03-25T18:30:48.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.975+0000 7fd9684e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:30:48.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.975+0000 7fd9684e6640 1 -- 172.21.15.3:0/2864256460 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd960148650 con 0x7fd960143f90 2024-03-25T18:30:48.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.975+0000 7fd9684e6640 10 client.4457.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:48.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.975+0000 7fd9684e6640 10 monclient: _renew_subs 2024-03-25T18:30:48.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.975+0000 7fd9684e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:30:48.978 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.975+0000 7fd9684e6640 1 -- 172.21.15.3:0/2864256460 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd960145fb0 con 0x7fd960143f90 2024-03-25T18:30:48.978 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.975+0000 7fd9684e6640 1 librados: init done 2024-03-25T18:30:48.978 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.976+0000 7fd9684e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:48.978 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.976+0000 7fd9684e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:48.978 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.976+0000 7fd9684e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:30:48.978 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:48.976+0000 7fd9684e6640 1 -- 172.21.15.3:0/2864256460 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd96006dbd0 con 0x7fd960143f90 2024-03-25T18:30:49.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.286+0000 7fd956ffd640 1 -- 172.21.15.3:0/2864256460 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9580020e0 con 0x7fd960143f90 2024-03-25T18:30:49.288 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.286+0000 7fd956ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:49.288 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.286+0000 7fd956ffd640 1 -- 172.21.15.3:0/2864256460 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd958004fb0 con 0x7fd960143f90 2024-03-25T18:30:49.288 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.286+0000 7fd956ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:49.288 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.286+0000 7fd956ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:49.288 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.286+0000 7fd956ffd640 10 monclient: dump: 2024-03-25T18:30:49.288 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:49.288 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:49.288 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:49.288 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:49.288 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:49.288 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:49.288 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:49.288 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:49.289 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:49.289 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:49.289 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.286+0000 7fd964a7f640 4 set_mon_vals no callback set 2024-03-25T18:30:49.289 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.287+0000 7fd956ffd640 1 -- 172.21.15.3:0/2864256460 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7fd95801af80 con 0x7fd960143f90 2024-03-25T18:30:49.289 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.288+0000 7fd966282640 10 monclient: get_auth_request con 0x7fd94406ded0 auth_method 0 2024-03-25T18:30:49.289 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.288+0000 7fd956ffd640 10 client.4457.objecter ms_handle_connect 0x7fd94406ded0 2024-03-25T18:30:49.294 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.293+0000 7fd956ffd640 1 -- 172.21.15.3:0/2864256460 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7fd958054b90 con 0x7fd960143f90 2024-03-25T18:30:49.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.293+0000 7fd956ffd640 10 client.4457.objecter ms_dispatch 0x7fd960000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:49.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.293+0000 7fd956ffd640 3 client.4457.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:49.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.293+0000 7fd956ffd640 3 client.4457.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:49.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.293+0000 7fd956ffd640 20 client.4457.objecter dump_active .. 0 homeless 2024-03-25T18:30:49.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.293+0000 7fd956ffd640 1 -- 172.21.15.3:0/2864256460 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd9580504d0 con 0x7fd960143f90 2024-03-25T18:30:49.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.294+0000 7fd956ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:49.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.294+0000 7fd956ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:49.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.552+0000 7fd9684e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:49.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.552+0000 7fd9684e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:49.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.552+0000 7fd9684e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:30:49.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.552+0000 7fd9684e6640 1 -- 172.21.15.3:0/2864256460 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9601dcae0 con 0x7fd960143f90 2024-03-25T18:30:49.581 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.580+0000 7fd956ffd640 1 -- 172.21.15.3:0/2864256460 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7fd9580525d0 con 0x7fd960143f90 2024-03-25T18:30:49.582 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.580+0000 7fd956ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:49.582 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.580+0000 7fd956ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:49.582 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:30:49.582 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:49.585 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.583+0000 7fd9684e6640 10 librados: watch_flush enter 2024-03-25T18:30:49.585 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.583+0000 7fd9684e6640 10 librados: watch_flush exit 2024-03-25T18:30:49.585 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.583+0000 7fd9684e6640 20 client.4457.objecter shutdown clearing up homeless session... 2024-03-25T18:30:49.585 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.583+0000 7fd9684e6640 10 client.4457.objecter successfully canceled tick 2024-03-25T18:30:49.585 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.583+0000 7fd9684e6640 1 -- 172.21.15.3:0/2864256460 >> v1:172.21.15.3:6800/4195140966 conn(0x7fd94406ded0 legacy=0x7fd944070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:49.586 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.583+0000 7fd9684e6640 10 monclient: shutdown 2024-03-25T18:30:49.586 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.583+0000 7fd9684e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:49.586 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.583+0000 7fd9684e6640 1 -- 172.21.15.3:0/2864256460 >> v1:172.21.15.3:6790/0 conn(0x7fd960143f90 legacy=0x7fd9600b8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:49.586 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.583+0000 7fd9684e6640 1 -- 172.21.15.3:0/2864256460 shutdown_connections 2024-03-25T18:30:49.586 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.583+0000 7fd9684e6640 1 -- 172.21.15.3:0/2864256460 >> 172.21.15.3:0/2864256460 conn(0x7fd9600617a0 msgr2=0x7fd960143650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:49.586 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.584+0000 7fd9684e6640 1 -- 172.21.15.3:0/2864256460 shutdown_connections 2024-03-25T18:30:49.586 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.584+0000 7fd9684e6640 1 -- 172.21.15.3:0/2864256460 wait complete. 2024-03-25T18:30:49.586 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:49.584+0000 7fd9684e6640 1 librados: shutdown 2024-03-25T18:30:49.596 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:56.402 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:56.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.521+0000 7fc6ea9ba640 10 monclient: get_monmap_and_config 2024-03-25T18:30:56.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.521+0000 7fc6ea9ba640 10 monclient: build_initial_monmap 2024-03-25T18:30:56.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.521+0000 7fc6ea9ba640 10 monclient: monmap: 2024-03-25T18:30:56.522 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:30:56.522 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:56.522 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:30:56.522478+0000 2024-03-25T18:30:56.523 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:30:56.522478+0000 2024-03-25T18:30:56.523 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:56.523 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:56.523 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:30:56.523 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:30:56.523 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:30:56.523 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:56.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.522+0000 7fc6ea9ba640 1 Processor -- start 2024-03-25T18:30:56.524 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.522+0000 7fc6ea9ba640 1 -- start start 2024-03-25T18:30:56.524 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.522+0000 7fc6ea9ba640 10 monclient: init 2024-03-25T18:30:56.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.523+0000 7fc6ea9ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:56.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.523+0000 7fc6ea9ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:30:56.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.523+0000 7fc6ea9ba640 10 monclient(hunting): picked mon.noname-b con 0x7fc6e40b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:30:56.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.523+0000 7fc6ea9ba640 10 monclient(hunting): picked mon.noname-c con 0x7fc6e40b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:30:56.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.523+0000 7fc6ea9ba640 10 monclient(hunting): picked mon.noname-a con 0x7fc6e413c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:30:56.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.523+0000 7fc6ea9ba640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e40ab0e0 con 0x7fc6e413c480 2024-03-25T18:30:56.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.523+0000 7fc6ea9ba640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e40ab3c0 con 0x7fc6e40b4670 2024-03-25T18:30:56.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.523+0000 7fc6ea9ba640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e40a9f30 con 0x7fc6e40b4ff0 2024-03-25T18:30:56.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.523+0000 7fc6ea9ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:56.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.523+0000 7fc6ea9ba640 10 monclient(hunting): authenticate will time out at 7710.783804s 2024-03-25T18:30:56.529 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.528+0000 7fc6e37fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc6e40b4670 0x7fc6e41425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:46510/0 (socket says 172.21.15.3:46510) 2024-03-25T18:30:56.529 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.528+0000 7fc6e37fe640 1 -- 172.21.15.3:0/2031005986 learned_addr learned my addr 172.21.15.3:0/2031005986 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:30:56.545 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.544+0000 7fc6e2ffd640 1 -- 172.21.15.3:0/2031005986 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 533246388 0 0) 0x7fc6e40ab0e0 con 0x7fc6e413c480 2024-03-25T18:30:56.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.544+0000 7fc6e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:56.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.544+0000 7fc6e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:56.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.544+0000 7fc6e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:56.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.544+0000 7fc6e2ffd640 10 monclient(hunting): my global_id is 4497 2024-03-25T18:30:56.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.544+0000 7fc6e2ffd640 1 -- 172.21.15.3:0/2031005986 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6cc003520 con 0x7fc6e413c480 2024-03-25T18:30:56.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.545+0000 7fc6e2ffd640 1 -- 172.21.15.3:0/2031005986 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262924205 0 0) 0x7fc6e40ab3c0 con 0x7fc6e40b4670 2024-03-25T18:30:56.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.545+0000 7fc6e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:56.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.545+0000 7fc6e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:56.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.545+0000 7fc6e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:56.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.545+0000 7fc6e2ffd640 10 monclient(hunting): my global_id is 4450 2024-03-25T18:30:56.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.545+0000 7fc6e2ffd640 1 -- 172.21.15.3:0/2031005986 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6e40ab0e0 con 0x7fc6e40b4670 2024-03-25T18:30:56.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.565+0000 7fc6e2ffd640 1 -- 172.21.15.3:0/2031005986 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 7288870 0 0) 0x7fc6e40a9f30 con 0x7fc6e40b4ff0 2024-03-25T18:30:56.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.565+0000 7fc6e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:56.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.565+0000 7fc6e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:56.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.565+0000 7fc6e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:56.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.565+0000 7fc6e2ffd640 10 monclient(hunting): my global_id is 4460 2024-03-25T18:30:56.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.565+0000 7fc6e2ffd640 1 -- 172.21.15.3:0/2031005986 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6e40ab3c0 con 0x7fc6e40b4ff0 2024-03-25T18:30:56.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.570+0000 7fc6e2ffd640 1 -- 172.21.15.3:0/2031005986 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3990351130 0 0) 0x7fc6e40ab0e0 con 0x7fc6e40b4670 2024-03-25T18:30:56.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.571+0000 7fc6e2ffd640 1 -- 172.21.15.3:0/2031005986 >> v1:172.21.15.3:6790/0 conn(0x7fc6e40b4ff0 legacy=0x7fc6e413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:56.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.571+0000 7fc6e2ffd640 1 -- 172.21.15.3:0/2031005986 >> v1:172.21.15.3:6789/0 conn(0x7fc6e413c480 legacy=0x7fc6e414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:56.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.571+0000 7fc6e2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:56.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.571+0000 7fc6e2ffd640 1 monclient: found mon.noname-c 2024-03-25T18:30:56.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.571+0000 7fc6e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:56.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.571+0000 7fc6e2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:30:56.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.571+0000 7fc6e2ffd640 1 -- 172.21.15.3:0/2031005986 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e40a9b70 con 0x7fc6e40b4670 2024-03-25T18:30:56.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.571+0000 7fc6e2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:30:56.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.571+0000 7fc6e2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:56.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.571+0000 7fc6e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:56.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.571+0000 7fc6e2ffd640 1 -- 172.21.15.3:0/2031005986 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc6d8001670 con 0x7fc6e40b4670 2024-03-25T18:30:56.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.571+0000 7fc6e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:56.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.571+0000 7fc6e2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:56.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.571+0000 7fc6e2ffd640 10 monclient: dump: 2024-03-25T18:30:56.573 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:56.573 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:56.573 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:56.573 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:56.574 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:56.574 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:56.574 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:56.574 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:56.574 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:56.574 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:56.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.571+0000 7fc6ea9ba640 5 monclient: authenticate success, global_id 4450 2024-03-25T18:30:56.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.571+0000 7fc6ea9ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:56.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.607+0000 7fc6e2ffd640 1 -- 172.21.15.3:0/2031005986 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6d80017d0 con 0x7fc6e40b4670 2024-03-25T18:30:56.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.607+0000 7fc6e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:56.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.607+0000 7fc6e2ffd640 1 -- 172.21.15.3:0/2031005986 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc6d8001be0 con 0x7fc6e40b4670 2024-03-25T18:30:56.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.607+0000 7fc6ea9ba640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:56.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.607+0000 7fc6ea9ba640 4 set_mon_vals no callback set 2024-03-25T18:30:56.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.607+0000 7fc6e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:56.609 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.607+0000 7fc6e2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:56.609 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.607+0000 7fc6e2ffd640 10 monclient: dump: 2024-03-25T18:30:56.609 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:56.609 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:56.609 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:56.609 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:56.609 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:56.609 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:56.609 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:56.609 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:56.609 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:56.609 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:56.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.607+0000 7fc6ea9ba640 10 monclient: shutdown 2024-03-25T18:30:56.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.607+0000 7fc6ea9ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:56.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.607+0000 7fc6ea9ba640 1 -- 172.21.15.3:0/2031005986 >> v1:172.21.15.16:6789/0 conn(0x7fc6e40b4670 legacy=0x7fc6e41425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:56.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.607+0000 7fc6ea9ba640 1 -- 172.21.15.3:0/2031005986 shutdown_connections 2024-03-25T18:30:56.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.608+0000 7fc6ea9ba640 1 -- 172.21.15.3:0/2031005986 >> 172.21.15.3:0/2031005986 conn(0x7fc6e40617a0 msgr2=0x7fc6e4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:56.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.608+0000 7fc6ea9ba640 1 -- 172.21.15.3:0/2031005986 shutdown_connections 2024-03-25T18:30:56.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.608+0000 7fc6ea9ba640 1 -- 172.21.15.3:0/2031005986 wait complete. 2024-03-25T18:30:56.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.608+0000 7fc6ea9ba640 10 monclient: build_initial_monmap 2024-03-25T18:30:56.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.608+0000 7fc6ea9ba640 10 monclient: monmap: 2024-03-25T18:30:56.612 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:30:56.613 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:56.613 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:30:56.613 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:30:56.613 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:56.613 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:30:56.613 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:30:56.613 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:30:56.613 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:30:56.613 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:56.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.609+0000 7fc6ea9ba640 1 librados: starting msgr at 2024-03-25T18:30:56.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.609+0000 7fc6ea9ba640 1 librados: starting objecter 2024-03-25T18:30:56.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.609+0000 7fc6ea9ba640 1 Processor -- start 2024-03-25T18:30:56.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.609+0000 7fc6ea9ba640 1 -- start start 2024-03-25T18:30:56.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.609+0000 7fc6ea9ba640 1 librados: setting wanted keys 2024-03-25T18:30:56.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.609+0000 7fc6ea9ba640 1 librados: calling monclient init 2024-03-25T18:30:56.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.609+0000 7fc6ea9ba640 10 monclient: init 2024-03-25T18:30:56.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.610+0000 7fc6ea9ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:56.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.610+0000 7fc6ea9ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:30:56.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.610+0000 7fc6ea9ba640 10 monclient(hunting): picked mon.noname-c con 0x7fc6e40b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:30:56.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.610+0000 7fc6ea9ba640 10 monclient(hunting): picked mon.noname-a con 0x7fc6e40b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:30:56.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.610+0000 7fc6ea9ba640 10 monclient(hunting): picked mon.noname-b con 0x7fc6e413c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:30:56.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.610+0000 7fc6ea9ba640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e413f7b0 con 0x7fc6e40b4ff0 2024-03-25T18:30:56.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.610+0000 7fc6ea9ba640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e41448e0 con 0x7fc6e413c480 2024-03-25T18:30:56.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.610+0000 7fc6ea9ba640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e4144bc0 con 0x7fc6e40b4670 2024-03-25T18:30:56.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.610+0000 7fc6ea9ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:56.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.610+0000 7fc6ea9ba640 10 monclient(hunting): authenticate will time out at 7710.870831s 2024-03-25T18:30:56.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.614+0000 7fc6e37fe640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fc6e40b4ff0 0x7fc6e413d350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:42502/0 (socket says 172.21.15.3:42502) 2024-03-25T18:30:56.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.614+0000 7fc6e37fe640 1 -- 172.21.15.3:0/283887487 learned_addr learned my addr 172.21.15.3:0/283887487 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:30:56.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.617+0000 7fc6e0ff9640 10 client.?.objecter ms_handle_connect 0x7fc6e413c480 2024-03-25T18:30:56.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.617+0000 7fc6e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:56.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.618+0000 7fc6e0ff9640 10 client.?.objecter ms_handle_connect 0x7fc6e40b4ff0 2024-03-25T18:30:56.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.618+0000 7fc6e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:56.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.627+0000 7fc6e0ff9640 1 -- 172.21.15.3:0/283887487 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1197184396 0 0) 0x7fc6e413f7b0 con 0x7fc6e40b4ff0 2024-03-25T18:30:56.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.628+0000 7fc6e0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:56.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.628+0000 7fc6e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:56.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.628+0000 7fc6e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:56.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.628+0000 7fc6e0ff9640 10 monclient(hunting): my global_id is 4500 2024-03-25T18:30:56.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.628+0000 7fc6e0ff9640 1 -- 172.21.15.3:0/283887487 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6d0004eb0 con 0x7fc6e40b4ff0 2024-03-25T18:30:56.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.628+0000 7fc6e0ff9640 1 -- 172.21.15.3:0/283887487 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528912674 0 0) 0x7fc6e41448e0 con 0x7fc6e413c480 2024-03-25T18:30:56.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.628+0000 7fc6e0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:56.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.628+0000 7fc6e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:56.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.628+0000 7fc6e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:56.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.628+0000 7fc6e0ff9640 10 monclient(hunting): my global_id is 4453 2024-03-25T18:30:56.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.628+0000 7fc6e0ff9640 1 -- 172.21.15.3:0/283887487 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6e413f7b0 con 0x7fc6e413c480 2024-03-25T18:30:56.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.632+0000 7fc6e0ff9640 10 client.?.objecter ms_handle_connect 0x7fc6e40b4670 2024-03-25T18:30:56.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.632+0000 7fc6e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:56.639 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.638+0000 7fc6e0ff9640 1 -- 172.21.15.3:0/283887487 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3051927395 0 0) 0x7fc6d0004eb0 con 0x7fc6e40b4ff0 2024-03-25T18:30:56.639 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.638+0000 7fc6e0ff9640 1 -- 172.21.15.3:0/283887487 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6e41448e0 con 0x7fc6e40b4ff0 2024-03-25T18:30:56.639 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.638+0000 7fc6e0ff9640 1 -- 172.21.15.3:0/283887487 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc6d8001c20 con 0x7fc6e40b4ff0 2024-03-25T18:30:56.639 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.638+0000 7fc6e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:56.639 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.638+0000 7fc6e0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:56.640 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.638+0000 7fc6e0ff9640 10 monclient(hunting): dump: 2024-03-25T18:30:56.640 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:56.640 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:56.640 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:56.640 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:56.640 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:56.640 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:56.640 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:56.640 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:56.640 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:56.640 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:56.641 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.638+0000 7fc6e0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:56.649 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.647+0000 7fc6e0ff9640 1 -- 172.21.15.3:0/283887487 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3593987315 0 0) 0x7fc6e41448e0 con 0x7fc6e40b4ff0 2024-03-25T18:30:56.649 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.647+0000 7fc6e0ff9640 1 -- 172.21.15.3:0/283887487 >> v1:172.21.15.3:6790/0 conn(0x7fc6e40b4670 legacy=0x7fc6e413cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:56.649 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.647+0000 7fc6e0ff9640 1 -- 172.21.15.3:0/283887487 >> v1:172.21.15.16:6789/0 conn(0x7fc6e413c480 legacy=0x7fc6e41e2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:56.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.648+0000 7fc6e0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:30:56.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.648+0000 7fc6e0ff9640 1 monclient: found mon.a 2024-03-25T18:30:56.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.648+0000 7fc6e0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:56.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.648+0000 7fc6e0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:30:56.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.648+0000 7fc6e0ff9640 1 -- 172.21.15.3:0/283887487 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e4146600 con 0x7fc6e40b4ff0 2024-03-25T18:30:56.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.648+0000 7fc6e0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:30:56.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.648+0000 7fc6e0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:30:56.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.648+0000 7fc6e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:56.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.648+0000 7fc6ea9ba640 5 monclient: authenticate success, global_id 4500 2024-03-25T18:30:56.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.648+0000 7fc6ea9ba640 10 monclient: _renew_subs 2024-03-25T18:30:56.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.648+0000 7fc6ea9ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:30:56.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.648+0000 7fc6ea9ba640 1 -- 172.21.15.3:0/283887487 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6e413eca0 con 0x7fc6e40b4ff0 2024-03-25T18:30:56.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.648+0000 7fc6ea9ba640 10 client.4500.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:56.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.648+0000 7fc6ea9ba640 10 monclient: _renew_subs 2024-03-25T18:30:56.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.648+0000 7fc6ea9ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:30:56.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.648+0000 7fc6ea9ba640 1 -- 172.21.15.3:0/283887487 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6e4143860 con 0x7fc6e40b4ff0 2024-03-25T18:30:56.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.648+0000 7fc6ea9ba640 1 librados: init done 2024-03-25T18:30:56.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.649+0000 7fc6ea9ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:56.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.649+0000 7fc6ea9ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:56.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.649+0000 7fc6ea9ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:30:56.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.649+0000 7fc6ea9ba640 1 -- 172.21.15.3:0/283887487 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6e40b7b40 con 0x7fc6e40b4ff0 2024-03-25T18:30:56.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.963+0000 7fc6e0ff9640 1 -- 172.21.15.3:0/283887487 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6d8004a90 con 0x7fc6e40b4ff0 2024-03-25T18:30:56.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.963+0000 7fc6e0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:56.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.963+0000 7fc6e0ff9640 1 -- 172.21.15.3:0/283887487 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc6d8004de0 con 0x7fc6e40b4ff0 2024-03-25T18:30:56.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.963+0000 7fc6e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:56.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.963+0000 7fc6e0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:56.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.963+0000 7fc6e0ff9640 10 monclient: dump: 2024-03-25T18:30:56.964 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:30:56.964 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:30:56.965 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:56.965 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:30:56.965 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:56.965 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:30:56.965 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:30:56.965 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:30:56.965 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:30:56.965 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:30:56.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.963+0000 7fc6e27fc640 4 set_mon_vals no callback set 2024-03-25T18:30:56.966 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.963+0000 7fc6e0ff9640 1 -- 172.21.15.3:0/283887487 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7fc6d8005000 con 0x7fc6e40b4ff0 2024-03-25T18:30:56.966 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.964+0000 7fc6e3fff640 10 monclient: get_auth_request con 0x7fc6d006dd90 auth_method 0 2024-03-25T18:30:56.966 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.964+0000 7fc6e0ff9640 10 client.4500.objecter ms_handle_connect 0x7fc6d006dd90 2024-03-25T18:30:56.983 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.982+0000 7fc6e0ff9640 1 -- 172.21.15.3:0/283887487 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7fc6d8054990 con 0x7fc6e40b4ff0 2024-03-25T18:30:56.983 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.982+0000 7fc6e0ff9640 10 client.4500.objecter ms_dispatch 0x7fc6e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:56.983 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.982+0000 7fc6e0ff9640 3 client.4500.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:56.983 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.982+0000 7fc6e0ff9640 3 client.4500.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:56.984 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.982+0000 7fc6e0ff9640 20 client.4500.objecter dump_active .. 0 homeless 2024-03-25T18:30:56.984 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.982+0000 7fc6e0ff9640 1 -- 172.21.15.3:0/283887487 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc6d80504a0 con 0x7fc6e40b4ff0 2024-03-25T18:30:56.984 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.982+0000 7fc6e0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:56.984 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:56.982+0000 7fc6e0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:57.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:57.209+0000 7fc6ea9ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:57.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:57.209+0000 7fc6ea9ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:57.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:57.209+0000 7fc6ea9ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:30:57.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:57.209+0000 7fc6ea9ba640 1 -- 172.21.15.3:0/283887487 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6e406dbd0 con 0x7fc6e40b4ff0 2024-03-25T18:30:57.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:57.238+0000 7fc6e0ff9640 1 -- 172.21.15.3:0/283887487 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7fc6d80523d0 con 0x7fc6e40b4ff0 2024-03-25T18:30:57.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:57.238+0000 7fc6e0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:57.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:57.238+0000 7fc6e0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:57.240 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:30:57.240 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:57.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:57.241+0000 7fc6ea9ba640 10 librados: watch_flush enter 2024-03-25T18:30:57.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:57.241+0000 7fc6ea9ba640 10 librados: watch_flush exit 2024-03-25T18:30:57.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:57.241+0000 7fc6ea9ba640 20 client.4500.objecter shutdown clearing up homeless session... 2024-03-25T18:30:57.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:57.241+0000 7fc6ea9ba640 10 client.4500.objecter successfully canceled tick 2024-03-25T18:30:57.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:57.241+0000 7fc6ea9ba640 1 -- 172.21.15.3:0/283887487 >> v1:172.21.15.3:6800/4195140966 conn(0x7fc6d006dd90 legacy=0x7fc6d0070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:57.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:57.241+0000 7fc6ea9ba640 10 monclient: shutdown 2024-03-25T18:30:57.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:57.241+0000 7fc6ea9ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:57.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:57.241+0000 7fc6ea9ba640 1 -- 172.21.15.3:0/283887487 >> v1:172.21.15.3:6789/0 conn(0x7fc6e40b4ff0 legacy=0x7fc6e413d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:57.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:57.241+0000 7fc6ea9ba640 1 -- 172.21.15.3:0/283887487 shutdown_connections 2024-03-25T18:30:57.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:57.241+0000 7fc6ea9ba640 1 -- 172.21.15.3:0/283887487 >> 172.21.15.3:0/283887487 conn(0x7fc6e40617a0 msgr2=0x7fc6e4140670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:57.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:57.241+0000 7fc6ea9ba640 1 -- 172.21.15.3:0/283887487 shutdown_connections 2024-03-25T18:30:57.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:57.241+0000 7fc6ea9ba640 1 -- 172.21.15.3:0/283887487 wait complete. 2024-03-25T18:30:57.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:30:57.241+0000 7fc6ea9ba640 1 librados: shutdown 2024-03-25T18:30:57.254 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:02.113 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:06:15.842 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:02.172 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:06:15.901 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:02.213 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:06:15.942 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:02.227 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:06:15.919 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:02.241 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:06:15.933 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:02.249 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:06:15.978 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:02.254 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:06:15.984 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:02.270 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:06:15.962 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:02.292 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:06:16.021 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:04.062 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:04.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.179+0000 7f5bd892b640 10 monclient: get_monmap_and_config 2024-03-25T18:31:04.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.179+0000 7f5bd892b640 10 monclient: build_initial_monmap 2024-03-25T18:31:04.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.179+0000 7f5bd892b640 10 monclient: monmap: 2024-03-25T18:31:04.181 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:31:04.181 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:04.181 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:31:04.180353+0000 2024-03-25T18:31:04.181 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:31:04.180353+0000 2024-03-25T18:31:04.181 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:04.181 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:04.182 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:31:04.182 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:31:04.182 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:31:04.182 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:04.182 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.179+0000 7f5bd892b640 1 Processor -- start 2024-03-25T18:31:04.182 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.180+0000 7f5bd892b640 1 -- start start 2024-03-25T18:31:04.182 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.180+0000 7f5bd892b640 10 monclient: init 2024-03-25T18:31:04.183 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.180+0000 7f5bd892b640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:04.183 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.180+0000 7f5bd892b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:31:04.183 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.180+0000 7f5bd892b640 10 monclient(hunting): picked mon.noname-c con 0x7f5bd40b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:31:04.183 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.180+0000 7f5bd892b640 10 monclient(hunting): picked mon.noname-b con 0x7f5bd40b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:31:04.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.180+0000 7f5bd892b640 10 monclient(hunting): picked mon.noname-a con 0x7f5bd413c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:31:04.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.181+0000 7f5bd892b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd40ab0e0 con 0x7f5bd413c480 2024-03-25T18:31:04.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.181+0000 7f5bd892b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd40ab3c0 con 0x7f5bd40b4ff0 2024-03-25T18:31:04.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.181+0000 7f5bd892b640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd40a9f30 con 0x7f5bd40b4670 2024-03-25T18:31:04.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.181+0000 7f5bd892b640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:04.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.181+0000 7f5bd892b640 10 monclient(hunting): authenticate will time out at 7718.441404s 2024-03-25T18:31:04.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.189+0000 7f5bd259c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5bd40b4ff0 0x7f5bd413bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:60336/0 (socket says 172.21.15.3:60336) 2024-03-25T18:31:04.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.189+0000 7f5bd259c640 1 -- 172.21.15.3:0/334257792 learned_addr learned my addr 172.21.15.3:0/334257792 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:31:04.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.202+0000 7f5bd1d9b640 1 -- 172.21.15.3:0/334257792 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2680702553 0 0) 0x7f5bd40ab0e0 con 0x7f5bd413c480 2024-03-25T18:31:04.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.202+0000 7f5bd1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:31:04.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.202+0000 7f5bd1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:04.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.202+0000 7f5bd1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:04.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.202+0000 7f5bd1d9b640 10 monclient(hunting): my global_id is 4503 2024-03-25T18:31:04.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.202+0000 7f5bd1d9b640 1 -- 172.21.15.3:0/334257792 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bc0003520 con 0x7f5bd413c480 2024-03-25T18:31:04.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.205+0000 7f5bd1d9b640 1 -- 172.21.15.3:0/334257792 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683566366 0 0) 0x7f5bd40ab3c0 con 0x7f5bd40b4ff0 2024-03-25T18:31:04.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.205+0000 7f5bd1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:31:04.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.205+0000 7f5bd1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:04.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.205+0000 7f5bd1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:04.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.205+0000 7f5bd1d9b640 10 monclient(hunting): my global_id is 4456 2024-03-25T18:31:04.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.205+0000 7f5bd1d9b640 1 -- 172.21.15.3:0/334257792 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bd40ab0e0 con 0x7f5bd40b4ff0 2024-03-25T18:31:04.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.217+0000 7f5bd1d9b640 1 -- 172.21.15.3:0/334257792 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1740430791 0 0) 0x7f5bd40a9f30 con 0x7f5bd40b4670 2024-03-25T18:31:04.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.217+0000 7f5bd1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:31:04.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.217+0000 7f5bd1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:04.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.217+0000 7f5bd1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:04.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.217+0000 7f5bd1d9b640 10 monclient(hunting): my global_id is 4466 2024-03-25T18:31:04.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.217+0000 7f5bd1d9b640 1 -- 172.21.15.3:0/334257792 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bd40ab3c0 con 0x7f5bd40b4670 2024-03-25T18:31:04.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.221+0000 7f5bd1d9b640 1 -- 172.21.15.3:0/334257792 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3853023300 0 0) 0x7f5bc0003520 con 0x7f5bd413c480 2024-03-25T18:31:04.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.221+0000 7f5bd1d9b640 1 -- 172.21.15.3:0/334257792 >> v1:172.21.15.3:6790/0 conn(0x7f5bd40b4670 legacy=0x7f5bd41425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:04.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.221+0000 7f5bd1d9b640 1 -- 172.21.15.3:0/334257792 >> v1:172.21.15.16:6789/0 conn(0x7f5bd40b4ff0 legacy=0x7f5bd413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:04.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.221+0000 7f5bd1d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:31:04.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.221+0000 7f5bd1d9b640 1 monclient: found mon.noname-a 2024-03-25T18:31:04.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.221+0000 7f5bd1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:04.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.221+0000 7f5bd1d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:31:04.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.221+0000 7f5bd1d9b640 1 -- 172.21.15.3:0/334257792 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd40a9b70 con 0x7f5bd413c480 2024-03-25T18:31:04.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.221+0000 7f5bd1d9b640 10 monclient: _finish_auth 0 2024-03-25T18:31:04.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.221+0000 7f5bd1d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:31:04.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.221+0000 7f5bd1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:04.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.221+0000 7f5bd1d9b640 1 -- 172.21.15.3:0/334257792 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f5bcc001a10 con 0x7f5bd413c480 2024-03-25T18:31:04.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.221+0000 7f5bd892b640 5 monclient: authenticate success, global_id 4503 2024-03-25T18:31:04.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.221+0000 7f5bd892b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:04.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.221+0000 7f5bd1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:04.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.221+0000 7f5bd1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:04.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.221+0000 7f5bd1d9b640 10 monclient: dump: 2024-03-25T18:31:04.224 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:04.224 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:04.224 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:04.224 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:04.225 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:04.225 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:04.225 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:04.225 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:04.225 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:04.225 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:04.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.221+0000 7f5bd892b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:04.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.237+0000 7f5bd1d9b640 1 -- 172.21.15.3:0/334257792 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bcc001b70 con 0x7f5bd413c480 2024-03-25T18:31:04.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.237+0000 7f5bd1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:04.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.237+0000 7f5bd1d9b640 1 -- 172.21.15.3:0/334257792 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f5bcc001f80 con 0x7f5bd413c480 2024-03-25T18:31:04.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.237+0000 7f5bd892b640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:04.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.237+0000 7f5bd892b640 4 set_mon_vals no callback set 2024-03-25T18:31:04.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.237+0000 7f5bd1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:04.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.237+0000 7f5bd1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:04.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.237+0000 7f5bd1d9b640 10 monclient: dump: 2024-03-25T18:31:04.239 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:04.239 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:04.239 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:04.240 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:04.240 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:04.240 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:04.240 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:04.240 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:04.240 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:04.240 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:04.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.238+0000 7f5bd892b640 10 monclient: shutdown 2024-03-25T18:31:04.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.238+0000 7f5bd892b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:04.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.238+0000 7f5bd892b640 1 -- 172.21.15.3:0/334257792 >> v1:172.21.15.3:6789/0 conn(0x7f5bd413c480 legacy=0x7f5bd414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:04.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.238+0000 7f5bd892b640 1 -- 172.21.15.3:0/334257792 shutdown_connections 2024-03-25T18:31:04.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.238+0000 7f5bd892b640 1 -- 172.21.15.3:0/334257792 >> 172.21.15.3:0/334257792 conn(0x7f5bd40617a0 msgr2=0x7f5bd4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:04.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.238+0000 7f5bd892b640 1 -- 172.21.15.3:0/334257792 shutdown_connections 2024-03-25T18:31:04.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.239+0000 7f5bd892b640 1 -- 172.21.15.3:0/334257792 wait complete. 2024-03-25T18:31:04.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.240+0000 7f5bd892b640 10 monclient: build_initial_monmap 2024-03-25T18:31:04.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.240+0000 7f5bd892b640 10 monclient: monmap: 2024-03-25T18:31:04.242 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:31:04.242 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:04.242 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:31:04.243 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:31:04.243 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:04.243 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:31:04.243 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:31:04.243 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:31:04.243 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:31:04.243 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:04.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.240+0000 7f5bd892b640 1 librados: starting msgr at 2024-03-25T18:31:04.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.240+0000 7f5bd892b640 1 librados: starting objecter 2024-03-25T18:31:04.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.240+0000 7f5bd892b640 1 Processor -- start 2024-03-25T18:31:04.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.240+0000 7f5bd892b640 1 -- start start 2024-03-25T18:31:04.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.240+0000 7f5bd892b640 1 librados: setting wanted keys 2024-03-25T18:31:04.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.240+0000 7f5bd892b640 1 librados: calling monclient init 2024-03-25T18:31:04.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.240+0000 7f5bd892b640 10 monclient: init 2024-03-25T18:31:04.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.241+0000 7f5bd892b640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:04.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.241+0000 7f5bd892b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:31:04.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.241+0000 7f5bd892b640 10 monclient(hunting): picked mon.noname-a con 0x7f5bd40b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:31:04.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.241+0000 7f5bd892b640 10 monclient(hunting): picked mon.noname-b con 0x7f5bd40b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:31:04.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.241+0000 7f5bd892b640 10 monclient(hunting): picked mon.noname-c con 0x7f5bd413c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:31:04.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.241+0000 7f5bd892b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd40aefb0 con 0x7f5bd40b4670 2024-03-25T18:31:04.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.241+0000 7f5bd892b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd40b0220 con 0x7f5bd40b4ff0 2024-03-25T18:31:04.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.241+0000 7f5bd892b640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd40b0500 con 0x7f5bd413c480 2024-03-25T18:31:04.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.241+0000 7f5bd892b640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:04.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.241+0000 7f5bd892b640 10 monclient(hunting): authenticate will time out at 7718.501574s 2024-03-25T18:31:04.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.248+0000 7f5bd2d9d640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f5bd413c480 0x7f5bd41e2380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:53534/0 (socket says 172.21.15.3:53534) 2024-03-25T18:31:04.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.248+0000 7f5bd2d9d640 1 -- 172.21.15.3:0/4004531051 learned_addr learned my addr 172.21.15.3:0/4004531051 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:31:04.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.249+0000 7f5bcb7fe640 10 client.?.objecter ms_handle_connect 0x7f5bd413c480 2024-03-25T18:31:04.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.249+0000 7f5bcb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:04.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.257+0000 7f5bcb7fe640 10 client.?.objecter ms_handle_connect 0x7f5bd40b4670 2024-03-25T18:31:04.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.257+0000 7f5bcb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:04.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.258+0000 7f5bcb7fe640 10 client.?.objecter ms_handle_connect 0x7f5bd40b4ff0 2024-03-25T18:31:04.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.258+0000 7f5bcb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:04.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.264+0000 7f5bcb7fe640 1 -- 172.21.15.3:0/4004531051 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 354461957 0 0) 0x7f5bd40b0500 con 0x7f5bd413c480 2024-03-25T18:31:04.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.264+0000 7f5bcb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:04.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.264+0000 7f5bcb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:04.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.264+0000 7f5bcb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:04.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.264+0000 7f5bcb7fe640 10 monclient(hunting): my global_id is 4469 2024-03-25T18:31:04.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.264+0000 7f5bcb7fe640 1 -- 172.21.15.3:0/4004531051 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb4004eb0 con 0x7f5bd413c480 2024-03-25T18:31:04.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.271+0000 7f5bcb7fe640 1 -- 172.21.15.3:0/4004531051 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1662079992 0 0) 0x7f5bd40aefb0 con 0x7f5bd40b4670 2024-03-25T18:31:04.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.271+0000 7f5bcb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:04.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.271+0000 7f5bcb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:04.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.271+0000 7f5bcb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:04.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.271+0000 7f5bcb7fe640 10 monclient(hunting): my global_id is 4506 2024-03-25T18:31:04.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.271+0000 7f5bcb7fe640 1 -- 172.21.15.3:0/4004531051 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bd40b0500 con 0x7f5bd40b4670 2024-03-25T18:31:04.279 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.278+0000 7f5bcb7fe640 1 -- 172.21.15.3:0/4004531051 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1203068793 0 0) 0x7f5bd40b0220 con 0x7f5bd40b4ff0 2024-03-25T18:31:04.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.278+0000 7f5bcb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:04.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.278+0000 7f5bcb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:04.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.278+0000 7f5bcb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:04.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.278+0000 7f5bcb7fe640 10 monclient(hunting): my global_id is 4459 2024-03-25T18:31:04.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.278+0000 7f5bcb7fe640 1 -- 172.21.15.3:0/4004531051 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bd40aefb0 con 0x7f5bd40b4ff0 2024-03-25T18:31:04.282 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.280+0000 7f5bcb7fe640 1 -- 172.21.15.3:0/4004531051 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3837632416 0 0) 0x7f5bb4004eb0 con 0x7f5bd413c480 2024-03-25T18:31:04.282 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.281+0000 7f5bcb7fe640 1 -- 172.21.15.3:0/4004531051 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bd40b0220 con 0x7f5bd413c480 2024-03-25T18:31:04.282 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.281+0000 7f5bcb7fe640 1 -- 172.21.15.3:0/4004531051 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f5bcc001ec0 con 0x7f5bd413c480 2024-03-25T18:31:04.282 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.281+0000 7f5bcb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:04.282 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.281+0000 7f5bcb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:04.282 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.281+0000 7f5bcb7fe640 10 monclient(hunting): dump: 2024-03-25T18:31:04.282 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:04.282 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:04.282 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:04.283 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:04.283 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:04.283 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:04.283 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:04.283 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:04.283 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:04.283 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:04.283 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.281+0000 7f5bcb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:04.300 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.299+0000 7f5bcb7fe640 1 -- 172.21.15.3:0/4004531051 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1514054968 0 0) 0x7f5bd40aefb0 con 0x7f5bd40b4ff0 2024-03-25T18:31:04.300 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.299+0000 7f5bcb7fe640 1 -- 172.21.15.3:0/4004531051 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bb4004eb0 con 0x7f5bd40b4ff0 2024-03-25T18:31:04.300 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.299+0000 7f5bcb7fe640 1 -- 172.21.15.3:0/4004531051 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f5bbc001830 con 0x7f5bd40b4ff0 2024-03-25T18:31:04.300 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.299+0000 7f5bcb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:04.300 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.299+0000 7f5bcb7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:04.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.299+0000 7f5bcb7fe640 10 monclient(hunting): dump: 2024-03-25T18:31:04.301 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:04.301 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:04.301 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:04.301 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:04.301 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:04.301 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:04.301 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:04.301 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:04.301 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:04.302 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:04.309 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.308+0000 7f5bcb7fe640 1 -- 172.21.15.3:0/4004531051 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4206566970 0 0) 0x7f5bd40b0500 con 0x7f5bd40b4670 2024-03-25T18:31:04.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.308+0000 7f5bcb7fe640 1 -- 172.21.15.3:0/4004531051 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bd40aefb0 con 0x7f5bd40b4670 2024-03-25T18:31:04.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.308+0000 7f5bcb7fe640 1 -- 172.21.15.3:0/4004531051 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f5bc4001970 con 0x7f5bd40b4670 2024-03-25T18:31:04.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.308+0000 7f5bcb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:04.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.308+0000 7f5bcb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:04.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.308+0000 7f5bcb7fe640 10 monclient(hunting): dump: 2024-03-25T18:31:04.310 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:04.310 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:04.310 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:04.310 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:04.311 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:04.311 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:04.311 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:04.311 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:04.311 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:04.311 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:04.312 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.309+0000 7f5bcb7fe640 1 -- 172.21.15.3:0/4004531051 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3919736109 0 0) 0x7f5bd40b0220 con 0x7f5bd413c480 2024-03-25T18:31:04.312 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.310+0000 7f5bcb7fe640 1 -- 172.21.15.3:0/4004531051 >> v1:172.21.15.16:6789/0 conn(0x7f5bd40b4ff0 legacy=0x7f5bd40b0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:04.312 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.310+0000 7f5bcb7fe640 1 -- 172.21.15.3:0/4004531051 >> v1:172.21.15.3:6789/0 conn(0x7f5bd40b4670 legacy=0x7f5bd40ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:04.312 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.310+0000 7f5bcb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:31:04.312 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.310+0000 7f5bcb7fe640 1 monclient: found mon.c 2024-03-25T18:31:04.312 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.310+0000 7f5bcb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:04.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.310+0000 7f5bcb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:31:04.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.310+0000 7f5bcb7fe640 1 -- 172.21.15.3:0/4004531051 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd40b07e0 con 0x7f5bd413c480 2024-03-25T18:31:04.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.310+0000 7f5bcb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:31:04.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.310+0000 7f5bcb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:31:04.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.310+0000 7f5bcb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:04.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.310+0000 7f5bd892b640 5 monclient: authenticate success, global_id 4469 2024-03-25T18:31:04.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.310+0000 7f5bd892b640 10 monclient: _renew_subs 2024-03-25T18:31:04.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.310+0000 7f5bd892b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:31:04.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.310+0000 7f5bd892b640 1 -- 172.21.15.3:0/4004531051 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bd40aebb0 con 0x7f5bd413c480 2024-03-25T18:31:04.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.310+0000 7f5bd892b640 10 client.4469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:04.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.310+0000 7f5bd892b640 10 monclient: _renew_subs 2024-03-25T18:31:04.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.310+0000 7f5bd892b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:31:04.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.310+0000 7f5bd892b640 1 -- 172.21.15.3:0/4004531051 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bd40b14f0 con 0x7f5bd413c480 2024-03-25T18:31:04.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.310+0000 7f5bd892b640 1 librados: init done 2024-03-25T18:31:04.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.311+0000 7f5bd892b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:04.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.311+0000 7f5bd892b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:04.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.311+0000 7f5bd892b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:31:04.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.311+0000 7f5bd892b640 1 -- 172.21.15.3:0/4004531051 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bd406dbd0 con 0x7f5bd413c480 2024-03-25T18:31:04.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.632+0000 7f5bcb7fe640 1 -- 172.21.15.3:0/4004531051 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bcc003c20 con 0x7f5bd413c480 2024-03-25T18:31:04.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.632+0000 7f5bcb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:04.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.632+0000 7f5bcb7fe640 1 -- 172.21.15.3:0/4004531051 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f5bcc002100 con 0x7f5bd413c480 2024-03-25T18:31:04.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.632+0000 7f5bcb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:04.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.632+0000 7f5bcb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:04.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.632+0000 7f5bcb7fe640 10 monclient: dump: 2024-03-25T18:31:04.634 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:04.634 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:04.634 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:04.635 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:04.635 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:04.635 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:04.635 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:04.635 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:04.635 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:04.635 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:04.635 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.632+0000 7f5bd159a640 4 set_mon_vals no callback set 2024-03-25T18:31:04.636 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.633+0000 7f5bcb7fe640 1 -- 172.21.15.3:0/4004531051 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f5bcc0040f0 con 0x7f5bd413c480 2024-03-25T18:31:04.636 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.634+0000 7f5bd259c640 10 monclient: get_auth_request con 0x7f5bb406e0f0 auth_method 0 2024-03-25T18:31:04.636 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.634+0000 7f5bcb7fe640 10 client.4469.objecter ms_handle_connect 0x7f5bb406e0f0 2024-03-25T18:31:04.641 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.640+0000 7f5bcb7fe640 1 -- 172.21.15.3:0/4004531051 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f5bcc054b20 con 0x7f5bd413c480 2024-03-25T18:31:04.642 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.640+0000 7f5bcb7fe640 10 client.4469.objecter ms_dispatch 0x7f5bd4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:04.642 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.640+0000 7f5bcb7fe640 3 client.4469.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:04.642 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.640+0000 7f5bcb7fe640 3 client.4469.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:04.642 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.640+0000 7f5bcb7fe640 20 client.4469.objecter dump_active .. 0 homeless 2024-03-25T18:31:04.642 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.640+0000 7f5bcb7fe640 1 -- 172.21.15.3:0/4004531051 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5bcc050630 con 0x7f5bd413c480 2024-03-25T18:31:04.642 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.640+0000 7f5bcb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:04.642 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.640+0000 7f5bcb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:04.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.890+0000 7f5bd892b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:04.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.890+0000 7f5bd892b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:04.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.890+0000 7f5bd892b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:31:04.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.890+0000 7f5bd892b640 1 -- 172.21.15.3:0/4004531051 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5bd40aed40 con 0x7f5bd413c480 2024-03-25T18:31:04.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.932+0000 7f5bcb7fe640 1 -- 172.21.15.3:0/4004531051 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f5bcc052560 con 0x7f5bd413c480 2024-03-25T18:31:04.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.932+0000 7f5bcb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:04.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.932+0000 7f5bcb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:04.934 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:31:04.934 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:04.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.935+0000 7f5bd892b640 10 librados: watch_flush enter 2024-03-25T18:31:04.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.935+0000 7f5bd892b640 10 librados: watch_flush exit 2024-03-25T18:31:04.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.935+0000 7f5bd892b640 20 client.4469.objecter shutdown clearing up homeless session... 2024-03-25T18:31:04.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.935+0000 7f5bd892b640 10 client.4469.objecter successfully canceled tick 2024-03-25T18:31:04.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.935+0000 7f5bd892b640 1 -- 172.21.15.3:0/4004531051 >> v1:172.21.15.3:6800/4195140966 conn(0x7f5bb406e0f0 legacy=0x7f5bb40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:04.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.935+0000 7f5bd892b640 10 monclient: shutdown 2024-03-25T18:31:04.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.935+0000 7f5bd892b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:04.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.935+0000 7f5bd892b640 1 -- 172.21.15.3:0/4004531051 >> v1:172.21.15.3:6790/0 conn(0x7f5bd413c480 legacy=0x7f5bd41e2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:04.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.935+0000 7f5bd892b640 1 -- 172.21.15.3:0/4004531051 shutdown_connections 2024-03-25T18:31:04.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.935+0000 7f5bd892b640 1 -- 172.21.15.3:0/4004531051 >> 172.21.15.3:0/4004531051 conn(0x7f5bd40617a0 msgr2=0x7f5bd40b2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:04.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.935+0000 7f5bd892b640 1 -- 172.21.15.3:0/4004531051 shutdown_connections 2024-03-25T18:31:04.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.935+0000 7f5bd892b640 1 -- 172.21.15.3:0/4004531051 wait complete. 2024-03-25T18:31:04.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:04.935+0000 7f5bd892b640 1 librados: shutdown 2024-03-25T18:31:04.948 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:11.756 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:11.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.866+0000 7f795bff4640 10 monclient: get_monmap_and_config 2024-03-25T18:31:11.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.866+0000 7f795bff4640 10 monclient: build_initial_monmap 2024-03-25T18:31:11.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.866+0000 7f795bff4640 10 monclient: monmap: 2024-03-25T18:31:11.868 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:31:11.868 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:11.868 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:31:11.867989+0000 2024-03-25T18:31:11.868 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:31:11.867989+0000 2024-03-25T18:31:11.868 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:11.868 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:11.868 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:31:11.868 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:31:11.869 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:31:11.869 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:11.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.867+0000 7f795bff4640 1 Processor -- start 2024-03-25T18:31:11.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.867+0000 7f795bff4640 1 -- start start 2024-03-25T18:31:11.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.867+0000 7f795bff4640 10 monclient: init 2024-03-25T18:31:11.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.868+0000 7f795bff4640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:11.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.868+0000 7f795bff4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:31:11.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.868+0000 7f795bff4640 10 monclient(hunting): picked mon.noname-a con 0x7f79540b4e70 addr v1:172.21.15.3:6789/0 2024-03-25T18:31:11.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.868+0000 7f795bff4640 10 monclient(hunting): picked mon.noname-c con 0x7f79540b44b0 addr v1:172.21.15.16:6789/0 2024-03-25T18:31:11.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.868+0000 7f795bff4640 10 monclient(hunting): picked mon.noname-b con 0x7f79540b1b00 addr v1:172.21.15.3:6790/0 2024-03-25T18:31:11.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.868+0000 7f795bff4640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79540a96f0 con 0x7f79540b4e70 2024-03-25T18:31:11.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.868+0000 7f795bff4640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79540ab0e0 con 0x7f79540b44b0 2024-03-25T18:31:11.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.868+0000 7f795bff4640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7954145f70 con 0x7f79540b1b00 2024-03-25T18:31:11.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.868+0000 7f795bff4640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:11.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.868+0000 7f795bff4640 10 monclient(hunting): authenticate will time out at 7726.128630s 2024-03-25T18:31:11.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.868+0000 7f795a591640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f79540b1b00 0x7f795414dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:42982/0 (socket says 172.21.15.3:42982) 2024-03-25T18:31:11.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.868+0000 7f795a591640 1 -- 172.21.15.3:0/13131980 learned_addr learned my addr 172.21.15.3:0/13131980 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:31:11.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.909+0000 7f7958d8e640 1 -- 172.21.15.3:0/13131980 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1011308007 0 0) 0x7f79540a96f0 con 0x7f79540b4e70 2024-03-25T18:31:11.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.909+0000 7f7958d8e640 10 monclient(hunting): _negotiate 2024-03-25T18:31:11.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.909+0000 7f7958d8e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:11.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.909+0000 7f7958d8e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:11.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.909+0000 7f7958d8e640 10 monclient(hunting): my global_id is 4509 2024-03-25T18:31:11.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.909+0000 7f7958d8e640 1 -- 172.21.15.3:0/13131980 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f793c003520 con 0x7f79540b4e70 2024-03-25T18:31:11.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.913+0000 7f7958d8e640 1 -- 172.21.15.3:0/13131980 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3567718610 0 0) 0x7f793c003520 con 0x7f79540b4e70 2024-03-25T18:31:11.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.913+0000 7f7958d8e640 1 -- 172.21.15.3:0/13131980 >> v1:172.21.15.3:6790/0 conn(0x7f79540b1b00 legacy=0x7f795414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:11.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.913+0000 7f7958d8e640 1 -- 172.21.15.3:0/13131980 >> v1:172.21.15.16:6789/0 conn(0x7f79540b44b0 legacy=0x7f7954145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:11.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.913+0000 7f7958d8e640 10 monclient: _finish_hunting 0 2024-03-25T18:31:11.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.913+0000 7f7958d8e640 1 monclient: found mon.noname-a 2024-03-25T18:31:11.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.913+0000 7f7958d8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:11.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.913+0000 7f7958d8e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:31:11.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.913+0000 7f7958d8e640 1 -- 172.21.15.3:0/13131980 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79540a9b80 con 0x7f79540b4e70 2024-03-25T18:31:11.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.913+0000 7f7958d8e640 10 monclient: _finish_auth 0 2024-03-25T18:31:11.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.913+0000 7f7958d8e640 10 monclient: _check_auth_tickets 2024-03-25T18:31:11.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.913+0000 7f7958d8e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:11.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.913+0000 7f7958d8e640 1 -- 172.21.15.3:0/13131980 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f794c001a10 con 0x7f79540b4e70 2024-03-25T18:31:11.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.913+0000 7f795bff4640 5 monclient: authenticate success, global_id 4509 2024-03-25T18:31:11.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.913+0000 7f795bff4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:11.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.913+0000 7f7958d8e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:11.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.913+0000 7f7958d8e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:11.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.913+0000 7f7958d8e640 10 monclient: dump: 2024-03-25T18:31:11.916 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:11.916 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:11.916 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:11.916 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:11.916 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:11.917 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:11.917 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:11.917 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:11.917 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:11.917 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:11.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.913+0000 7f795bff4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:11.931 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.927+0000 7f7958d8e640 1 -- 172.21.15.3:0/13131980 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f794c001b70 con 0x7f79540b4e70 2024-03-25T18:31:11.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.927+0000 7f7958d8e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:11.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.927+0000 7f7958d8e640 1 -- 172.21.15.3:0/13131980 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f794c001f80 con 0x7f79540b4e70 2024-03-25T18:31:11.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.927+0000 7f795bff4640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:11.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.927+0000 7f795bff4640 4 set_mon_vals no callback set 2024-03-25T18:31:11.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.927+0000 7f7958d8e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:11.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.927+0000 7f7958d8e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:11.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.927+0000 7f7958d8e640 10 monclient: dump: 2024-03-25T18:31:11.932 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:11.932 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:11.932 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:11.932 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:11.932 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:11.932 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:11.932 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:11.932 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:11.933 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:11.933 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:11.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.928+0000 7f795bff4640 10 monclient: shutdown 2024-03-25T18:31:11.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.928+0000 7f795bff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:11.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.928+0000 7f795bff4640 1 -- 172.21.15.3:0/13131980 >> v1:172.21.15.3:6789/0 conn(0x7f79540b4e70 legacy=0x7f795413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:11.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.928+0000 7f795bff4640 1 -- 172.21.15.3:0/13131980 shutdown_connections 2024-03-25T18:31:11.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.928+0000 7f795bff4640 1 -- 172.21.15.3:0/13131980 >> 172.21.15.3:0/13131980 conn(0x7f79540617a0 msgr2=0x7f7954061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:11.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.928+0000 7f795bff4640 1 -- 172.21.15.3:0/13131980 shutdown_connections 2024-03-25T18:31:11.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.928+0000 7f795bff4640 1 -- 172.21.15.3:0/13131980 wait complete. 2024-03-25T18:31:11.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.928+0000 7f795bff4640 10 monclient: build_initial_monmap 2024-03-25T18:31:11.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.928+0000 7f795bff4640 10 monclient: monmap: 2024-03-25T18:31:11.933 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:31:11.933 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:11.933 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:31:11.933 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:31:11.933 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:11.934 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:31:11.934 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:31:11.934 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:31:11.934 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:31:11.934 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:11.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.929+0000 7f795bff4640 1 librados: starting msgr at 2024-03-25T18:31:11.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.929+0000 7f795bff4640 1 librados: starting objecter 2024-03-25T18:31:11.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.929+0000 7f795bff4640 1 Processor -- start 2024-03-25T18:31:11.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.929+0000 7f795bff4640 1 -- start start 2024-03-25T18:31:11.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.929+0000 7f795bff4640 1 librados: setting wanted keys 2024-03-25T18:31:11.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.929+0000 7f795bff4640 1 librados: calling monclient init 2024-03-25T18:31:11.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.929+0000 7f795bff4640 10 monclient: init 2024-03-25T18:31:11.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.929+0000 7f795bff4640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:11.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.929+0000 7f795bff4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:31:11.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.929+0000 7f795bff4640 10 monclient(hunting): picked mon.noname-c con 0x7f79540b1b00 addr v1:172.21.15.3:6790/0 2024-03-25T18:31:11.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.929+0000 7f795bff4640 10 monclient(hunting): picked mon.noname-a con 0x7f79540b44b0 addr v1:172.21.15.3:6789/0 2024-03-25T18:31:11.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.929+0000 7f795bff4640 10 monclient(hunting): picked mon.noname-b con 0x7f79540b4e70 addr v1:172.21.15.16:6789/0 2024-03-25T18:31:11.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.929+0000 7f795bff4640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7954139100 con 0x7f79540b44b0 2024-03-25T18:31:11.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.929+0000 7f795bff4640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79541399f0 con 0x7f79540b4e70 2024-03-25T18:31:11.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.929+0000 7f795bff4640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79541406a0 con 0x7f79540b1b00 2024-03-25T18:31:11.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.929+0000 7f795bff4640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:11.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.929+0000 7f795bff4640 10 monclient(hunting): authenticate will time out at 7726.190273s 2024-03-25T18:31:11.947 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.946+0000 7f795958f640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f79540b44b0 0x7f7954135620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53074/0 (socket says 172.21.15.3:53074) 2024-03-25T18:31:11.947 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.946+0000 7f795958f640 1 -- 172.21.15.3:0/3151965023 learned_addr learned my addr 172.21.15.3:0/3151965023 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:31:11.949 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.948+0000 7f79527fc640 10 client.?.objecter ms_handle_connect 0x7f79540b44b0 2024-03-25T18:31:11.949 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.948+0000 7f79527fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:11.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.958+0000 7f79527fc640 1 -- 172.21.15.3:0/3151965023 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3817916149 0 0) 0x7f7954139100 con 0x7f79540b44b0 2024-03-25T18:31:11.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.958+0000 7f79527fc640 10 monclient(hunting): _negotiate 2024-03-25T18:31:11.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.958+0000 7f79527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:11.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.958+0000 7f79527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:11.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.958+0000 7f79527fc640 10 monclient(hunting): my global_id is 4512 2024-03-25T18:31:11.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.958+0000 7f79527fc640 1 -- 172.21.15.3:0/3151965023 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7940004eb0 con 0x7f79540b44b0 2024-03-25T18:31:11.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.964+0000 7f79527fc640 10 client.?.objecter ms_handle_connect 0x7f79540b4e70 2024-03-25T18:31:11.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.964+0000 7f79527fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:11.967 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.966+0000 7f79527fc640 10 client.?.objecter ms_handle_connect 0x7f79540b1b00 2024-03-25T18:31:11.967 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.966+0000 7f79527fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:11.981 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.979+0000 7f79527fc640 1 -- 172.21.15.3:0/3151965023 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3768921360 0 0) 0x7f7940004eb0 con 0x7f79540b44b0 2024-03-25T18:31:11.981 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.979+0000 7f79527fc640 1 -- 172.21.15.3:0/3151965023 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7954139100 con 0x7f79540b44b0 2024-03-25T18:31:11.981 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.979+0000 7f79527fc640 1 -- 172.21.15.3:0/3151965023 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7948001970 con 0x7f79540b44b0 2024-03-25T18:31:11.981 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.979+0000 7f79527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:11.981 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.979+0000 7f79527fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:11.981 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.979+0000 7f79527fc640 10 monclient(hunting): dump: 2024-03-25T18:31:11.981 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:11.981 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:11.981 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:11.981 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:11.981 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:11.981 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:11.981 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:11.982 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:11.982 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:11.982 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:11.982 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:11.979+0000 7f79527fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:12.049 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.048+0000 7f79527fc640 1 -- 172.21.15.3:0/3151965023 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4007948273 0 0) 0x7f7954139100 con 0x7f79540b44b0 2024-03-25T18:31:12.049 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.048+0000 7f79527fc640 1 -- 172.21.15.3:0/3151965023 >> v1:172.21.15.3:6790/0 conn(0x7f79540b1b00 legacy=0x7f7954137ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:12.049 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.048+0000 7f79527fc640 1 -- 172.21.15.3:0/3151965023 >> v1:172.21.15.16:6789/0 conn(0x7f79540b4e70 legacy=0x7f7954135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:12.049 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.048+0000 7f79527fc640 10 monclient: _finish_hunting 0 2024-03-25T18:31:12.049 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.048+0000 7f79527fc640 1 monclient: found mon.a 2024-03-25T18:31:12.049 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.048+0000 7f79527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:12.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.048+0000 7f79527fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:31:12.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.048+0000 7f79527fc640 1 -- 172.21.15.3:0/3151965023 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7954138c70 con 0x7f79540b44b0 2024-03-25T18:31:12.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.048+0000 7f79527fc640 10 monclient: _finish_auth 0 2024-03-25T18:31:12.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.048+0000 7f79527fc640 10 monclient: _check_auth_tickets 2024-03-25T18:31:12.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.048+0000 7f79527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:12.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.048+0000 7f795bff4640 5 monclient: authenticate success, global_id 4512 2024-03-25T18:31:12.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.048+0000 7f795bff4640 10 monclient: _renew_subs 2024-03-25T18:31:12.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.048+0000 7f795bff4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:31:12.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.048+0000 7f795bff4640 1 -- 172.21.15.3:0/3151965023 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79541e30d0 con 0x7f79540b44b0 2024-03-25T18:31:12.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.048+0000 7f795bff4640 10 client.4512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:12.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.048+0000 7f795bff4640 10 monclient: _renew_subs 2024-03-25T18:31:12.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.048+0000 7f795bff4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:31:12.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.048+0000 7f795bff4640 1 -- 172.21.15.3:0/3151965023 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f79541e3440 con 0x7f79540b44b0 2024-03-25T18:31:12.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.048+0000 7f795bff4640 1 librados: init done 2024-03-25T18:31:12.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.049+0000 7f795bff4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:12.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.049+0000 7f795bff4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:12.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.049+0000 7f795bff4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:31:12.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.049+0000 7f795bff4640 1 -- 172.21.15.3:0/3151965023 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f795404f730 con 0x7f79540b44b0 2024-03-25T18:31:12.422 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.420+0000 7f79527fc640 1 -- 172.21.15.3:0/3151965023 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7948003610 con 0x7f79540b44b0 2024-03-25T18:31:12.422 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.420+0000 7f79527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:12.422 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.420+0000 7f79527fc640 1 -- 172.21.15.3:0/3151965023 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7948003960 con 0x7f79540b44b0 2024-03-25T18:31:12.422 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.420+0000 7f79527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:12.422 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.420+0000 7f79527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:12.422 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.420+0000 7f79527fc640 10 monclient: dump: 2024-03-25T18:31:12.422 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:12.423 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:12.423 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:12.423 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:12.423 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:12.423 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:12.423 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:12.423 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:12.423 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:12.423 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:12.423 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.420+0000 7f7953fff640 4 set_mon_vals no callback set 2024-03-25T18:31:12.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.421+0000 7f79527fc640 1 -- 172.21.15.3:0/3151965023 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f7948003b40 con 0x7f79540b44b0 2024-03-25T18:31:12.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.422+0000 7f7959d90640 10 monclient: get_auth_request con 0x7f794006dc00 auth_method 0 2024-03-25T18:31:12.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.422+0000 7f79527fc640 10 client.4512.objecter ms_handle_connect 0x7f794006dc00 2024-03-25T18:31:12.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.428+0000 7f79527fc640 1 -- 172.21.15.3:0/3151965023 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f7948054810 con 0x7f79540b44b0 2024-03-25T18:31:12.430 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.428+0000 7f79527fc640 10 client.4512.objecter ms_dispatch 0x7f7954000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:12.430 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.428+0000 7f79527fc640 3 client.4512.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:12.430 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.428+0000 7f79527fc640 3 client.4512.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:12.430 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.428+0000 7f79527fc640 20 client.4512.objecter dump_active .. 0 homeless 2024-03-25T18:31:12.430 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.428+0000 7f79527fc640 1 -- 172.21.15.3:0/3151965023 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7948050320 con 0x7f79540b44b0 2024-03-25T18:31:12.430 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.428+0000 7f79527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:12.430 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.428+0000 7f79527fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:12.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.695+0000 7f795bff4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:12.697 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.695+0000 7f795bff4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:12.697 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.695+0000 7f795bff4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:31:12.697 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.695+0000 7f795bff4640 1 -- 172.21.15.3:0/3151965023 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79540aa090 con 0x7f79540b44b0 2024-03-25T18:31:12.725 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.724+0000 7f79527fc640 1 -- 172.21.15.3:0/3151965023 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f7948052250 con 0x7f79540b44b0 2024-03-25T18:31:12.725 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.724+0000 7f79527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:12.725 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.724+0000 7f79527fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:12.725 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:31:12.725 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:12.728 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.726+0000 7f795bff4640 10 librados: watch_flush enter 2024-03-25T18:31:12.728 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.726+0000 7f795bff4640 10 librados: watch_flush exit 2024-03-25T18:31:12.728 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.726+0000 7f795bff4640 20 client.4512.objecter shutdown clearing up homeless session... 2024-03-25T18:31:12.728 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.726+0000 7f795bff4640 10 client.4512.objecter successfully canceled tick 2024-03-25T18:31:12.728 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.726+0000 7f795bff4640 1 -- 172.21.15.3:0/3151965023 >> v1:172.21.15.3:6800/4195140966 conn(0x7f794006dc00 legacy=0x7f79400700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:12.728 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.726+0000 7f795bff4640 10 monclient: shutdown 2024-03-25T18:31:12.728 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.726+0000 7f795bff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:12.729 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.726+0000 7f795bff4640 1 -- 172.21.15.3:0/3151965023 >> v1:172.21.15.3:6789/0 conn(0x7f79540b44b0 legacy=0x7f7954135620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:12.729 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.726+0000 7f795bff4640 1 -- 172.21.15.3:0/3151965023 shutdown_connections 2024-03-25T18:31:12.729 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.726+0000 7f795bff4640 1 -- 172.21.15.3:0/3151965023 >> 172.21.15.3:0/3151965023 conn(0x7f79540617a0 msgr2=0x7f79540b6720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:12.729 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.726+0000 7f795bff4640 1 -- 172.21.15.3:0/3151965023 shutdown_connections 2024-03-25T18:31:12.729 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.727+0000 7f795bff4640 1 -- 172.21.15.3:0/3151965023 wait complete. 2024-03-25T18:31:12.729 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:12.727+0000 7f795bff4640 1 librados: shutdown 2024-03-25T18:31:12.741 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:19.547 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:19.661 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.659+0000 7f098533d640 10 monclient: get_monmap_and_config 2024-03-25T18:31:19.661 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.659+0000 7f098533d640 10 monclient: build_initial_monmap 2024-03-25T18:31:19.661 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.659+0000 7f098533d640 10 monclient: monmap: 2024-03-25T18:31:19.661 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:31:19.661 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:19.661 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:31:19.661045+0000 2024-03-25T18:31:19.661 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:31:19.661045+0000 2024-03-25T18:31:19.662 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:19.662 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:19.662 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:31:19.662 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:31:19.662 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:31:19.662 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:19.663 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.660+0000 7f098533d640 1 Processor -- start 2024-03-25T18:31:19.663 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.660+0000 7f098533d640 1 -- start start 2024-03-25T18:31:19.663 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.660+0000 7f098533d640 10 monclient: init 2024-03-25T18:31:19.663 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.661+0000 7f098533d640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:19.663 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.661+0000 7f098533d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:31:19.663 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.661+0000 7f098533d640 10 monclient(hunting): picked mon.noname-a con 0x7f09800b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:31:19.664 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.661+0000 7f098533d640 10 monclient(hunting): picked mon.noname-c con 0x7f09800b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:31:19.664 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.661+0000 7f098533d640 10 monclient(hunting): picked mon.noname-b con 0x7f098013c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:31:19.664 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.661+0000 7f098533d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09800ab0e0 con 0x7f09800b4ff0 2024-03-25T18:31:19.664 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.661+0000 7f098533d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09800ab3c0 con 0x7f09800b4670 2024-03-25T18:31:19.664 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.661+0000 7f098533d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09800a9f30 con 0x7f098013c480 2024-03-25T18:31:19.664 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.661+0000 7f098533d640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:19.664 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.661+0000 7f098533d640 10 monclient(hunting): authenticate will time out at 7733.921923s 2024-03-25T18:31:19.665 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.664+0000 7f0977fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f09800b4670 0x7f09801425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:56606/0 (socket says 172.21.15.3:56606) 2024-03-25T18:31:19.665 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.664+0000 7f0977fff640 1 -- 172.21.15.3:0/597649893 learned_addr learned my addr 172.21.15.3:0/597649893 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:31:19.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.677+0000 7f097e7fc640 1 -- 172.21.15.3:0/597649893 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307348827 0 0) 0x7f09800ab3c0 con 0x7f09800b4670 2024-03-25T18:31:19.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.677+0000 7f097e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:31:19.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.677+0000 7f097e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:19.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.677+0000 7f097e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:19.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.677+0000 7f097e7fc640 10 monclient(hunting): my global_id is 4465 2024-03-25T18:31:19.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.677+0000 7f097e7fc640 1 -- 172.21.15.3:0/597649893 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f096c003520 con 0x7f09800b4670 2024-03-25T18:31:19.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.692+0000 7f097e7fc640 1 -- 172.21.15.3:0/597649893 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4269030951 0 0) 0x7f096c003520 con 0x7f09800b4670 2024-03-25T18:31:19.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.692+0000 7f097e7fc640 1 -- 172.21.15.3:0/597649893 >> v1:172.21.15.3:6790/0 conn(0x7f098013c480 legacy=0x7f098014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:19.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.692+0000 7f097e7fc640 1 -- 172.21.15.3:0/597649893 >> v1:172.21.15.3:6789/0 conn(0x7f09800b4ff0 legacy=0x7f098013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:31:19.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.692+0000 7f097e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:31:19.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.692+0000 7f097e7fc640 1 monclient: found mon.noname-c 2024-03-25T18:31:19.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.692+0000 7f097e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:19.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.692+0000 7f097e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:31:19.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.692+0000 7f097e7fc640 1 -- 172.21.15.3:0/597649893 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09800a9b70 con 0x7f09800b4670 2024-03-25T18:31:19.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.692+0000 7f097e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:31:19.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.692+0000 7f097e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:31:19.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.692+0000 7f097e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:19.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.692+0000 7f097e7fc640 1 -- 172.21.15.3:0/597649893 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f0968001690 con 0x7f09800b4670 2024-03-25T18:31:19.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.692+0000 7f098533d640 5 monclient: authenticate success, global_id 4465 2024-03-25T18:31:19.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.692+0000 7f098533d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:19.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.692+0000 7f097e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:19.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.692+0000 7f097e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:19.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.692+0000 7f097e7fc640 10 monclient: dump: 2024-03-25T18:31:19.695 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:19.695 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:19.695 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:19.695 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:19.695 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:19.696 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:19.696 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:19.696 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:19.696 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:19.696 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:19.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.692+0000 7f098533d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:19.701 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.700+0000 7f097e7fc640 1 -- 172.21.15.3:0/597649893 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0968001860 con 0x7f09800b4670 2024-03-25T18:31:19.701 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.700+0000 7f097e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:19.701 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.700+0000 7f097e7fc640 1 -- 172.21.15.3:0/597649893 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f0968001cb0 con 0x7f09800b4670 2024-03-25T18:31:19.701 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.700+0000 7f098533d640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:19.701 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.700+0000 7f098533d640 4 set_mon_vals no callback set 2024-03-25T18:31:19.701 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.700+0000 7f097e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:19.701 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.700+0000 7f097e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:19.702 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.700+0000 7f097e7fc640 10 monclient: dump: 2024-03-25T18:31:19.702 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:19.702 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:19.702 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:19.702 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:19.702 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:19.702 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:19.702 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:19.703 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:19.703 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:19.703 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:19.705 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.700+0000 7f098533d640 10 monclient: shutdown 2024-03-25T18:31:19.705 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.700+0000 7f098533d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:19.705 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.700+0000 7f098533d640 1 -- 172.21.15.3:0/597649893 >> v1:172.21.15.16:6789/0 conn(0x7f09800b4670 legacy=0x7f09801425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:19.705 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.701+0000 7f098533d640 1 -- 172.21.15.3:0/597649893 shutdown_connections 2024-03-25T18:31:19.705 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.701+0000 7f098533d640 1 -- 172.21.15.3:0/597649893 >> 172.21.15.3:0/597649893 conn(0x7f09800617a0 msgr2=0x7f0980061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:19.705 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.701+0000 7f098533d640 1 -- 172.21.15.3:0/597649893 shutdown_connections 2024-03-25T18:31:19.705 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.701+0000 7f098533d640 1 -- 172.21.15.3:0/597649893 wait complete. 2024-03-25T18:31:19.705 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.701+0000 7f098533d640 10 monclient: build_initial_monmap 2024-03-25T18:31:19.705 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.701+0000 7f098533d640 10 monclient: monmap: 2024-03-25T18:31:19.705 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:31:19.705 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:19.706 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:31:19.706 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:31:19.706 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:19.706 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:31:19.706 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:31:19.706 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:31:19.706 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:31:19.706 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:19.706 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.702+0000 7f098533d640 1 librados: starting msgr at 2024-03-25T18:31:19.706 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.702+0000 7f098533d640 1 librados: starting objecter 2024-03-25T18:31:19.706 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.702+0000 7f098533d640 1 Processor -- start 2024-03-25T18:31:19.706 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.702+0000 7f098533d640 1 -- start start 2024-03-25T18:31:19.706 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.702+0000 7f098533d640 1 librados: setting wanted keys 2024-03-25T18:31:19.707 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.702+0000 7f098533d640 1 librados: calling monclient init 2024-03-25T18:31:19.707 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.702+0000 7f098533d640 10 monclient: init 2024-03-25T18:31:19.707 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.702+0000 7f098533d640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:19.707 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.703+0000 7f098533d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:31:19.707 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.703+0000 7f098533d640 10 monclient(hunting): picked mon.noname-b con 0x7f09800b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:31:19.707 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.703+0000 7f098533d640 10 monclient(hunting): picked mon.noname-a con 0x7f09800b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:31:19.707 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.703+0000 7f098533d640 10 monclient(hunting): picked mon.noname-c con 0x7f098013c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:31:19.707 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.703+0000 7f098533d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f098013df00 con 0x7f09800b4ff0 2024-03-25T18:31:19.707 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.703+0000 7f098533d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0980144dc0 con 0x7f09800b4670 2024-03-25T18:31:19.707 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.703+0000 7f098533d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f098013d000 con 0x7f098013c480 2024-03-25T18:31:19.707 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.703+0000 7f098533d640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:19.707 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.703+0000 7f098533d640 10 monclient(hunting): authenticate will time out at 7733.963433s 2024-03-25T18:31:19.718 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.717+0000 7f0977fff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f09800b4ff0 0x7f098013c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:39688/0 (socket says 172.21.15.3:39688) 2024-03-25T18:31:19.719 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.717+0000 7f0977fff640 1 -- 172.21.15.3:0/1212353316 learned_addr learned my addr 172.21.15.3:0/1212353316 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:31:19.722 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.721+0000 7f09777fe640 10 client.?.objecter ms_handle_connect 0x7f09800b4670 2024-03-25T18:31:19.722 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.721+0000 7f09777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:19.724 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.723+0000 7f09777fe640 10 client.?.objecter ms_handle_connect 0x7f098013c480 2024-03-25T18:31:19.724 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.723+0000 7f09777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:19.725 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.724+0000 7f09777fe640 10 client.?.objecter ms_handle_connect 0x7f09800b4ff0 2024-03-25T18:31:19.725 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.724+0000 7f09777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:19.732 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.730+0000 7f09777fe640 1 -- 172.21.15.3:0/1212353316 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2791606999 0 0) 0x7f0980144dc0 con 0x7f09800b4670 2024-03-25T18:31:19.732 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.730+0000 7f09777fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:19.732 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.730+0000 7f09777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:19.732 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.730+0000 7f09777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:19.732 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.730+0000 7f09777fe640 10 monclient(hunting): my global_id is 4468 2024-03-25T18:31:19.732 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.730+0000 7f09777fe640 1 -- 172.21.15.3:0/1212353316 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0960004eb0 con 0x7f09800b4670 2024-03-25T18:31:19.736 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.735+0000 7f09777fe640 1 -- 172.21.15.3:0/1212353316 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536449253 0 0) 0x7f098013df00 con 0x7f09800b4ff0 2024-03-25T18:31:19.736 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.735+0000 7f09777fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:19.736 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.735+0000 7f09777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:19.737 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.735+0000 7f09777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:19.737 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.735+0000 7f09777fe640 10 monclient(hunting): my global_id is 4515 2024-03-25T18:31:19.737 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.735+0000 7f09777fe640 1 -- 172.21.15.3:0/1212353316 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0980144dc0 con 0x7f09800b4ff0 2024-03-25T18:31:19.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.738+0000 7f09777fe640 1 -- 172.21.15.3:0/1212353316 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2466580114 0 0) 0x7f098013d000 con 0x7f098013c480 2024-03-25T18:31:19.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.738+0000 7f09777fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:19.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.738+0000 7f09777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:19.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.738+0000 7f09777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:19.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.738+0000 7f09777fe640 10 monclient(hunting): my global_id is 4478 2024-03-25T18:31:19.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.738+0000 7f09777fe640 1 -- 172.21.15.3:0/1212353316 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f098013df00 con 0x7f098013c480 2024-03-25T18:31:19.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.743+0000 7f09777fe640 1 -- 172.21.15.3:0/1212353316 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4269014619 0 0) 0x7f0960004eb0 con 0x7f09800b4670 2024-03-25T18:31:19.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.743+0000 7f09777fe640 1 -- 172.21.15.3:0/1212353316 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f098013d000 con 0x7f09800b4670 2024-03-25T18:31:19.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.743+0000 7f09777fe640 1 -- 172.21.15.3:0/1212353316 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f0970001870 con 0x7f09800b4670 2024-03-25T18:31:19.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.743+0000 7f09777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:19.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.743+0000 7f09777fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:19.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.743+0000 7f09777fe640 10 monclient(hunting): dump: 2024-03-25T18:31:19.745 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:19.745 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:19.745 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:19.745 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:19.745 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:19.746 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:19.746 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:19.746 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:19.746 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:19.746 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:19.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.743+0000 7f09777fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:19.752 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.750+0000 7f09777fe640 1 -- 172.21.15.3:0/1212353316 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 726535338 0 0) 0x7f0980144dc0 con 0x7f09800b4ff0 2024-03-25T18:31:19.752 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.750+0000 7f09777fe640 1 -- 172.21.15.3:0/1212353316 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0960004eb0 con 0x7f09800b4ff0 2024-03-25T18:31:19.752 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.750+0000 7f09777fe640 1 -- 172.21.15.3:0/1212353316 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f0968001b70 con 0x7f09800b4ff0 2024-03-25T18:31:19.752 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.750+0000 7f09777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:19.752 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.750+0000 7f09777fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:19.752 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.750+0000 7f09777fe640 10 monclient(hunting): dump: 2024-03-25T18:31:19.752 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:19.752 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:19.752 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:19.752 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:19.753 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:19.753 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:19.753 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:19.753 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:19.753 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:19.753 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:19.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.765+0000 7f09777fe640 1 -- 172.21.15.3:0/1212353316 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2413282616 0 0) 0x7f098013d000 con 0x7f09800b4670 2024-03-25T18:31:19.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.765+0000 7f09777fe640 1 -- 172.21.15.3:0/1212353316 >> v1:172.21.15.3:6790/0 conn(0x7f098013c480 legacy=0x7f09801e2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:19.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.765+0000 7f09777fe640 1 -- 172.21.15.3:0/1212353316 >> v1:172.21.15.3:6789/0 conn(0x7f09800b4ff0 legacy=0x7f098013c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:19.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.765+0000 7f09777fe640 10 monclient: _finish_hunting 0 2024-03-25T18:31:19.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.765+0000 7f09777fe640 1 monclient: found mon.b 2024-03-25T18:31:19.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.765+0000 7f09777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:19.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.765+0000 7f09777fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:31:19.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.765+0000 7f09777fe640 1 -- 172.21.15.3:0/1212353316 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f098013bff0 con 0x7f09800b4670 2024-03-25T18:31:19.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.765+0000 7f09777fe640 10 monclient: _finish_auth 0 2024-03-25T18:31:19.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.765+0000 7f09777fe640 10 monclient: _check_auth_tickets 2024-03-25T18:31:19.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.765+0000 7f09777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:19.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.765+0000 7f098533d640 5 monclient: authenticate success, global_id 4468 2024-03-25T18:31:19.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.765+0000 7f098533d640 10 monclient: _renew_subs 2024-03-25T18:31:19.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.765+0000 7f098533d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:31:19.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.765+0000 7f098533d640 1 -- 172.21.15.3:0/1212353316 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f098013f6b0 con 0x7f09800b4670 2024-03-25T18:31:19.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.765+0000 7f098533d640 10 client.4468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:19.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.765+0000 7f098533d640 10 monclient: _renew_subs 2024-03-25T18:31:19.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.765+0000 7f098533d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:31:19.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.765+0000 7f098533d640 1 -- 172.21.15.3:0/1212353316 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f09801463f0 con 0x7f09800b4670 2024-03-25T18:31:19.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.765+0000 7f098533d640 1 librados: init done 2024-03-25T18:31:19.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.766+0000 7f098533d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:19.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.766+0000 7f098533d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:19.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.766+0000 7f098533d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:31:19.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:19.766+0000 7f098533d640 1 -- 172.21.15.3:0/1212353316 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0980146740 con 0x7f09800b4670 2024-03-25T18:31:20.069 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.067+0000 7f09777fe640 1 -- 172.21.15.3:0/1212353316 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09700035f0 con 0x7f09800b4670 2024-03-25T18:31:20.069 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.067+0000 7f09777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:20.069 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.067+0000 7f09777fe640 1 -- 172.21.15.3:0/1212353316 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f0970001d80 con 0x7f09800b4670 2024-03-25T18:31:20.069 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.067+0000 7f09777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:20.069 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.067+0000 7f09777fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:20.069 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.067+0000 7f09777fe640 10 monclient: dump: 2024-03-25T18:31:20.069 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:20.069 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:20.069 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:20.070 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:20.070 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:20.070 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:20.070 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:20.070 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:20.070 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:20.070 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:20.070 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.067+0000 7f097dffb640 4 set_mon_vals no callback set 2024-03-25T18:31:20.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.068+0000 7f09777fe640 1 -- 172.21.15.3:0/1212353316 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f09700197f0 con 0x7f09800b4670 2024-03-25T18:31:20.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.069+0000 7f0977fff640 10 monclient: get_auth_request con 0x7f096006de70 auth_method 0 2024-03-25T18:31:20.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.069+0000 7f09777fe640 10 client.4468.objecter ms_handle_connect 0x7f096006de70 2024-03-25T18:31:20.075 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.073+0000 7f09777fe640 1 -- 172.21.15.3:0/1212353316 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f0970053600 con 0x7f09800b4670 2024-03-25T18:31:20.075 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.073+0000 7f09777fe640 10 client.4468.objecter ms_dispatch 0x7f0980000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:20.075 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.073+0000 7f09777fe640 3 client.4468.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:20.075 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.073+0000 7f09777fe640 3 client.4468.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:20.075 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.073+0000 7f09777fe640 20 client.4468.objecter dump_active .. 0 homeless 2024-03-25T18:31:20.076 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.075+0000 7f09777fe640 1 -- 172.21.15.3:0/1212353316 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f097004ef40 con 0x7f09800b4670 2024-03-25T18:31:20.076 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.075+0000 7f09777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:20.076 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.075+0000 7f09777fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:20.292 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.291+0000 7f098533d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:20.292 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.291+0000 7f098533d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:20.292 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.291+0000 7f098533d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:31:20.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.291+0000 7f098533d640 1 -- 172.21.15.3:0/1212353316 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09800b3ff0 con 0x7f09800b4670 2024-03-25T18:31:20.353 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.351+0000 7f09777fe640 1 -- 172.21.15.3:0/1212353316 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f0970051040 con 0x7f09800b4670 2024-03-25T18:31:20.353 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.351+0000 7f09777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:20.353 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.351+0000 7f09777fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:20.353 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:31:20.353 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:20.356 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.354+0000 7f098533d640 10 librados: watch_flush enter 2024-03-25T18:31:20.356 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.354+0000 7f098533d640 10 librados: watch_flush exit 2024-03-25T18:31:20.356 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.354+0000 7f098533d640 20 client.4468.objecter shutdown clearing up homeless session... 2024-03-25T18:31:20.356 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.354+0000 7f098533d640 10 client.4468.objecter successfully canceled tick 2024-03-25T18:31:20.356 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.354+0000 7f098533d640 1 -- 172.21.15.3:0/1212353316 >> v1:172.21.15.3:6800/4195140966 conn(0x7f096006de70 legacy=0x7f0960070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:20.356 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.354+0000 7f098533d640 10 monclient: shutdown 2024-03-25T18:31:20.356 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.354+0000 7f098533d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:20.356 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.354+0000 7f098533d640 1 -- 172.21.15.3:0/1212353316 >> v1:172.21.15.16:6789/0 conn(0x7f09800b4670 legacy=0x7f0980143680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:20.357 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.354+0000 7f098533d640 1 -- 172.21.15.3:0/1212353316 shutdown_connections 2024-03-25T18:31:20.357 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.354+0000 7f098533d640 1 -- 172.21.15.3:0/1212353316 >> 172.21.15.3:0/1212353316 conn(0x7f09800617a0 msgr2=0x7f09800b70a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:20.357 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.354+0000 7f098533d640 1 -- 172.21.15.3:0/1212353316 shutdown_connections 2024-03-25T18:31:20.357 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.354+0000 7f098533d640 1 -- 172.21.15.3:0/1212353316 wait complete. 2024-03-25T18:31:20.357 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:20.354+0000 7f098533d640 1 librados: shutdown 2024-03-25T18:31:20.366 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:22.675 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:06:36.404 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:22.722 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:06:36.414 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:22.733 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:06:36.425 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:22.735 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:06:36.464 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:22.765 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:06:36.458 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:22.772 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:06:36.501 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:22.773 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:06:36.502 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:22.788 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:06:36.517 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:22.818 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:06:36.547 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:27.173 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:27.290 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.289+0000 7fd077d15640 10 monclient: get_monmap_and_config 2024-03-25T18:31:27.291 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.289+0000 7fd077d15640 10 monclient: build_initial_monmap 2024-03-25T18:31:27.291 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.289+0000 7fd077d15640 10 monclient: monmap: 2024-03-25T18:31:27.291 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:31:27.291 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:27.291 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:31:27.290791+0000 2024-03-25T18:31:27.291 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:31:27.290791+0000 2024-03-25T18:31:27.291 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:27.291 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:27.291 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:31:27.291 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:31:27.291 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:31:27.291 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:27.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.290+0000 7fd077d15640 1 Processor -- start 2024-03-25T18:31:27.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.290+0000 7fd077d15640 1 -- start start 2024-03-25T18:31:27.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.290+0000 7fd077d15640 10 monclient: init 2024-03-25T18:31:27.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.291+0000 7fd077d15640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:27.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.291+0000 7fd077d15640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:31:27.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.291+0000 7fd077d15640 10 monclient(hunting): picked mon.noname-b con 0x7fd0700b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:31:27.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.291+0000 7fd077d15640 10 monclient(hunting): picked mon.noname-a con 0x7fd0700b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:31:27.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.291+0000 7fd077d15640 10 monclient(hunting): picked mon.noname-c con 0x7fd07013c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:31:27.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.291+0000 7fd077d15640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0700ab0e0 con 0x7fd0700b4670 2024-03-25T18:31:27.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.291+0000 7fd077d15640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0700ab3c0 con 0x7fd07013c480 2024-03-25T18:31:27.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.291+0000 7fd077d15640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0700a9f30 con 0x7fd0700b4ff0 2024-03-25T18:31:27.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.291+0000 7fd077d15640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:27.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.291+0000 7fd077d15640 10 monclient(hunting): authenticate will time out at 7741.551790s 2024-03-25T18:31:27.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.300+0000 7fd0752b0640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd0700b4670 0x7fd0701425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:44178/0 (socket says 172.21.15.3:44178) 2024-03-25T18:31:27.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.301+0000 7fd0752b0640 1 -- 172.21.15.3:0/3079453237 learned_addr learned my addr 172.21.15.3:0/3079453237 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:31:27.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.319+0000 7fd074aaf640 1 -- 172.21.15.3:0/3079453237 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364686688 0 0) 0x7fd0700ab0e0 con 0x7fd0700b4670 2024-03-25T18:31:27.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.319+0000 7fd074aaf640 10 monclient(hunting): _negotiate 2024-03-25T18:31:27.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.319+0000 7fd074aaf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:27.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.319+0000 7fd074aaf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:27.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.319+0000 7fd074aaf640 10 monclient(hunting): my global_id is 4518 2024-03-25T18:31:27.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.319+0000 7fd074aaf640 1 -- 172.21.15.3:0/3079453237 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd05c003520 con 0x7fd0700b4670 2024-03-25T18:31:27.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.326+0000 7fd074aaf640 1 -- 172.21.15.3:0/3079453237 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1411087221 0 0) 0x7fd0700a9f30 con 0x7fd0700b4ff0 2024-03-25T18:31:27.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.326+0000 7fd074aaf640 10 monclient(hunting): _negotiate 2024-03-25T18:31:27.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.326+0000 7fd074aaf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:27.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.326+0000 7fd074aaf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:27.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.326+0000 7fd074aaf640 10 monclient(hunting): my global_id is 4481 2024-03-25T18:31:27.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.326+0000 7fd074aaf640 1 -- 172.21.15.3:0/3079453237 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0700ab0e0 con 0x7fd0700b4ff0 2024-03-25T18:31:27.334 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.333+0000 7fd074aaf640 1 -- 172.21.15.3:0/3079453237 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 653459736 0 0) 0x7fd0700ab0e0 con 0x7fd0700b4ff0 2024-03-25T18:31:27.334 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.333+0000 7fd074aaf640 1 -- 172.21.15.3:0/3079453237 >> v1:172.21.15.16:6789/0 conn(0x7fd07013c480 legacy=0x7fd07014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:27.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.333+0000 7fd074aaf640 1 -- 172.21.15.3:0/3079453237 >> v1:172.21.15.3:6789/0 conn(0x7fd0700b4670 legacy=0x7fd0701425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:27.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.333+0000 7fd074aaf640 10 monclient: _finish_hunting 0 2024-03-25T18:31:27.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.333+0000 7fd074aaf640 1 monclient: found mon.noname-b 2024-03-25T18:31:27.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.333+0000 7fd074aaf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:27.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.333+0000 7fd074aaf640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-25T18:31:27.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.333+0000 7fd074aaf640 1 -- 172.21.15.3:0/3079453237 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0700a9b70 con 0x7fd0700b4ff0 2024-03-25T18:31:27.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.333+0000 7fd074aaf640 10 monclient: _finish_auth 0 2024-03-25T18:31:27.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.333+0000 7fd074aaf640 10 monclient: _check_auth_tickets 2024-03-25T18:31:27.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.333+0000 7fd074aaf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:27.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.333+0000 7fd074aaf640 1 -- 172.21.15.3:0/3079453237 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd06c001670 con 0x7fd0700b4ff0 2024-03-25T18:31:27.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.333+0000 7fd077d15640 5 monclient: authenticate success, global_id 4481 2024-03-25T18:31:27.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.333+0000 7fd077d15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:27.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.333+0000 7fd074aaf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:27.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.333+0000 7fd074aaf640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:27.336 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.333+0000 7fd074aaf640 10 monclient: dump: 2024-03-25T18:31:27.336 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:27.336 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:27.336 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:27.336 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:27.336 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:27.336 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:27.336 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:27.336 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:27.336 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:27.336 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:27.336 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.333+0000 7fd077d15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:27.362 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.361+0000 7fd074aaf640 1 -- 172.21.15.3:0/3079453237 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd06c0017d0 con 0x7fd0700b4ff0 2024-03-25T18:31:27.362 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.361+0000 7fd074aaf640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:27.362 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.361+0000 7fd074aaf640 1 -- 172.21.15.3:0/3079453237 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd06c001c20 con 0x7fd0700b4ff0 2024-03-25T18:31:27.363 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.361+0000 7fd077d15640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:27.363 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.361+0000 7fd077d15640 4 set_mon_vals no callback set 2024-03-25T18:31:27.363 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.361+0000 7fd074aaf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:27.363 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.361+0000 7fd074aaf640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:27.363 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.361+0000 7fd074aaf640 10 monclient: dump: 2024-03-25T18:31:27.363 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:27.363 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:27.363 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:27.363 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:27.363 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:27.363 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:27.363 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:27.363 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:27.363 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:27.364 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:27.365 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.362+0000 7fd077d15640 10 monclient: shutdown 2024-03-25T18:31:27.365 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.362+0000 7fd077d15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:27.365 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.362+0000 7fd077d15640 1 -- 172.21.15.3:0/3079453237 >> v1:172.21.15.3:6790/0 conn(0x7fd0700b4ff0 legacy=0x7fd07013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:27.365 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.362+0000 7fd077d15640 1 -- 172.21.15.3:0/3079453237 shutdown_connections 2024-03-25T18:31:27.365 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.362+0000 7fd077d15640 1 -- 172.21.15.3:0/3079453237 >> 172.21.15.3:0/3079453237 conn(0x7fd0700617a0 msgr2=0x7fd070061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:27.366 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.362+0000 7fd077d15640 1 -- 172.21.15.3:0/3079453237 shutdown_connections 2024-03-25T18:31:27.366 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.362+0000 7fd077d15640 1 -- 172.21.15.3:0/3079453237 wait complete. 2024-03-25T18:31:27.366 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.363+0000 7fd077d15640 10 monclient: build_initial_monmap 2024-03-25T18:31:27.366 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.363+0000 7fd077d15640 10 monclient: monmap: 2024-03-25T18:31:27.366 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:31:27.366 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:27.366 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:31:27.366 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:31:27.366 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:27.366 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:31:27.366 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:31:27.366 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:31:27.366 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:31:27.366 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:27.366 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.363+0000 7fd077d15640 1 librados: starting msgr at 2024-03-25T18:31:27.367 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.363+0000 7fd077d15640 1 librados: starting objecter 2024-03-25T18:31:27.367 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.363+0000 7fd077d15640 1 Processor -- start 2024-03-25T18:31:27.367 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.363+0000 7fd077d15640 1 -- start start 2024-03-25T18:31:27.367 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.363+0000 7fd077d15640 1 librados: setting wanted keys 2024-03-25T18:31:27.367 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.363+0000 7fd077d15640 1 librados: calling monclient init 2024-03-25T18:31:27.367 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.363+0000 7fd077d15640 10 monclient: init 2024-03-25T18:31:27.367 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.364+0000 7fd077d15640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:27.367 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.364+0000 7fd077d15640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:31:27.367 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.364+0000 7fd077d15640 10 monclient(hunting): picked mon.noname-c con 0x7fd0700b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:31:27.367 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.364+0000 7fd077d15640 10 monclient(hunting): picked mon.noname-b con 0x7fd0700b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:31:27.367 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.364+0000 7fd077d15640 10 monclient(hunting): picked mon.noname-a con 0x7fd07013c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:31:27.367 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.364+0000 7fd077d15640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd07013e000 con 0x7fd07013c480 2024-03-25T18:31:27.367 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.364+0000 7fd077d15640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd07013e2e0 con 0x7fd0700b4ff0 2024-03-25T18:31:27.367 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.364+0000 7fd077d15640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd07013f710 con 0x7fd0700b4670 2024-03-25T18:31:27.367 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.364+0000 7fd077d15640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:27.368 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.364+0000 7fd077d15640 10 monclient(hunting): authenticate will time out at 7741.624823s 2024-03-25T18:31:27.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.376+0000 7fd0752b0640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd0700b4ff0 0x7fd0701db620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:32808/0 (socket says 172.21.15.3:32808) 2024-03-25T18:31:27.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.376+0000 7fd0752b0640 1 -- 172.21.15.3:0/2134261988 learned_addr learned my addr 172.21.15.3:0/2134261988 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:31:27.379 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.378+0000 7fd0667fc640 10 client.?.objecter ms_handle_connect 0x7fd0700b4ff0 2024-03-25T18:31:27.379 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.378+0000 7fd0667fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:27.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.390+0000 7fd0667fc640 10 client.?.objecter ms_handle_connect 0x7fd0700b4670 2024-03-25T18:31:27.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.390+0000 7fd0667fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:27.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.390+0000 7fd0667fc640 1 -- 172.21.15.3:0/2134261988 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2653650702 0 0) 0x7fd07013e2e0 con 0x7fd0700b4ff0 2024-03-25T18:31:27.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.390+0000 7fd0667fc640 10 monclient(hunting): _negotiate 2024-03-25T18:31:27.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.390+0000 7fd0667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:27.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.390+0000 7fd0667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:27.392 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.390+0000 7fd0667fc640 10 monclient(hunting): my global_id is 4474 2024-03-25T18:31:27.392 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.390+0000 7fd0667fc640 1 -- 172.21.15.3:0/2134261988 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd054004eb0 con 0x7fd0700b4ff0 2024-03-25T18:31:27.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.394+0000 7fd0667fc640 1 -- 172.21.15.3:0/2134261988 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776950530 0 0) 0x7fd07013f710 con 0x7fd0700b4670 2024-03-25T18:31:27.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.394+0000 7fd0667fc640 10 monclient(hunting): _negotiate 2024-03-25T18:31:27.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.394+0000 7fd0667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:27.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.394+0000 7fd0667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:27.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.394+0000 7fd0667fc640 10 monclient(hunting): my global_id is 4484 2024-03-25T18:31:27.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.394+0000 7fd0667fc640 1 -- 172.21.15.3:0/2134261988 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd07013e2e0 con 0x7fd0700b4670 2024-03-25T18:31:27.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.405+0000 7fd0667fc640 10 client.?.objecter ms_handle_connect 0x7fd07013c480 2024-03-25T18:31:27.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.405+0000 7fd0667fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:27.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.409+0000 7fd0667fc640 1 -- 172.21.15.3:0/2134261988 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 715456439 0 0) 0x7fd054004eb0 con 0x7fd0700b4ff0 2024-03-25T18:31:27.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.409+0000 7fd0667fc640 1 -- 172.21.15.3:0/2134261988 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd07013f710 con 0x7fd0700b4ff0 2024-03-25T18:31:27.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.409+0000 7fd0667fc640 1 -- 172.21.15.3:0/2134261988 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd060001990 con 0x7fd0700b4ff0 2024-03-25T18:31:27.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.409+0000 7fd0667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:27.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.409+0000 7fd0667fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:27.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.409+0000 7fd0667fc640 10 monclient(hunting): dump: 2024-03-25T18:31:27.411 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:27.411 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:27.411 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:27.411 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:27.411 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:27.411 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:27.411 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:27.411 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:27.411 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:27.412 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:27.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.410+0000 7fd0667fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:27.416 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.414+0000 7fd0667fc640 1 -- 172.21.15.3:0/2134261988 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3313582278 0 0) 0x7fd07013e2e0 con 0x7fd0700b4670 2024-03-25T18:31:27.416 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.414+0000 7fd0667fc640 1 -- 172.21.15.3:0/2134261988 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd054004eb0 con 0x7fd0700b4670 2024-03-25T18:31:27.416 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.414+0000 7fd0667fc640 1 -- 172.21.15.3:0/2134261988 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd06c001b20 con 0x7fd0700b4670 2024-03-25T18:31:27.416 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.414+0000 7fd0667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:27.416 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.414+0000 7fd0667fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:27.416 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.414+0000 7fd0667fc640 10 monclient(hunting): dump: 2024-03-25T18:31:27.416 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:27.416 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:27.416 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:27.416 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:27.416 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:27.416 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:27.416 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:27.416 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:27.416 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:27.417 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:27.422 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.421+0000 7fd0667fc640 1 -- 172.21.15.3:0/2134261988 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 360021161 0 0) 0x7fd07013f710 con 0x7fd0700b4ff0 2024-03-25T18:31:27.423 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.421+0000 7fd0667fc640 1 -- 172.21.15.3:0/2134261988 >> v1:172.21.15.3:6790/0 conn(0x7fd0700b4670 legacy=0x7fd0701302c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:27.423 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.421+0000 7fd0667fc640 1 -- 172.21.15.3:0/2134261988 >> v1:172.21.15.3:6789/0 conn(0x7fd07013c480 legacy=0x7fd0701dbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:27.423 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.421+0000 7fd0667fc640 10 monclient: _finish_hunting 0 2024-03-25T18:31:27.423 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.421+0000 7fd0667fc640 1 monclient: found mon.b 2024-03-25T18:31:27.423 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.421+0000 7fd0667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:27.423 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.421+0000 7fd0667fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:31:27.423 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.421+0000 7fd0667fc640 1 -- 172.21.15.3:0/2134261988 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd07013f8a0 con 0x7fd0700b4ff0 2024-03-25T18:31:27.423 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.421+0000 7fd0667fc640 10 monclient: _finish_auth 0 2024-03-25T18:31:27.423 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.421+0000 7fd0667fc640 10 monclient: _check_auth_tickets 2024-03-25T18:31:27.423 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.421+0000 7fd0667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:27.423 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.421+0000 7fd077d15640 5 monclient: authenticate success, global_id 4474 2024-03-25T18:31:27.423 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.421+0000 7fd077d15640 10 monclient: _renew_subs 2024-03-25T18:31:27.423 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.421+0000 7fd077d15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:31:27.423 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.421+0000 7fd077d15640 1 -- 172.21.15.3:0/2134261988 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd07013cf10 con 0x7fd0700b4ff0 2024-03-25T18:31:27.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.421+0000 7fd077d15640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:27.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.421+0000 7fd077d15640 10 monclient: _renew_subs 2024-03-25T18:31:27.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.421+0000 7fd077d15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:31:27.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.421+0000 7fd077d15640 1 -- 172.21.15.3:0/2134261988 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd070143680 con 0x7fd0700b4ff0 2024-03-25T18:31:27.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.421+0000 7fd077d15640 1 librados: init done 2024-03-25T18:31:27.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.422+0000 7fd077d15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:27.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.422+0000 7fd077d15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:27.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.422+0000 7fd077d15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:31:27.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.422+0000 7fd077d15640 1 -- 172.21.15.3:0/2134261988 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd07006dbd0 con 0x7fd0700b4ff0 2024-03-25T18:31:27.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.740+0000 7fd0667fc640 1 -- 172.21.15.3:0/2134261988 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd0600047e0 con 0x7fd0700b4ff0 2024-03-25T18:31:27.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.740+0000 7fd0667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:27.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.740+0000 7fd0667fc640 1 -- 172.21.15.3:0/2134261988 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd060004b70 con 0x7fd0700b4ff0 2024-03-25T18:31:27.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.740+0000 7fd0667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:27.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.740+0000 7fd0667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:27.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.740+0000 7fd0667fc640 10 monclient: dump: 2024-03-25T18:31:27.742 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:27.743 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:27.743 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:27.743 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:27.743 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:27.743 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:27.743 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:27.743 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:27.743 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:27.743 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:27.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.740+0000 7fd067fff640 4 set_mon_vals no callback set 2024-03-25T18:31:27.744 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.742+0000 7fd0667fc640 1 -- 172.21.15.3:0/2134261988 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7fd06001ac00 con 0x7fd0700b4ff0 2024-03-25T18:31:27.744 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.742+0000 7fd075ab1640 10 monclient: get_auth_request con 0x7fd05406de80 auth_method 0 2024-03-25T18:31:27.744 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.742+0000 7fd0667fc640 10 client.4474.objecter ms_handle_connect 0x7fd05406de80 2024-03-25T18:31:27.748 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.747+0000 7fd0667fc640 1 -- 172.21.15.3:0/2134261988 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7fd060054900 con 0x7fd0700b4ff0 2024-03-25T18:31:27.748 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.747+0000 7fd0667fc640 10 client.4474.objecter ms_dispatch 0x7fd070000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:27.748 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.747+0000 7fd0667fc640 3 client.4474.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:27.749 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.747+0000 7fd0667fc640 3 client.4474.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:27.749 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.747+0000 7fd0667fc640 20 client.4474.objecter dump_active .. 0 homeless 2024-03-25T18:31:27.749 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.748+0000 7fd0667fc640 1 -- 172.21.15.3:0/2134261988 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd060050240 con 0x7fd0700b4ff0 2024-03-25T18:31:27.749 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.748+0000 7fd0667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:27.749 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.748+0000 7fd0667fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:27.975 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.974+0000 7fd077d15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:27.975 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.974+0000 7fd077d15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:27.975 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.974+0000 7fd077d15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:31:27.975 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:27.974+0000 7fd077d15640 1 -- 172.21.15.3:0/2134261988 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd0700b19b0 con 0x7fd0700b4ff0 2024-03-25T18:31:28.015 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:28.014+0000 7fd0667fc640 1 -- 172.21.15.3:0/2134261988 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7fd060052340 con 0x7fd0700b4ff0 2024-03-25T18:31:28.015 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:28.014+0000 7fd0667fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:28.015 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:28.014+0000 7fd0667fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:28.015 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:31:28.015 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:28.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:28.016+0000 7fd077d15640 10 librados: watch_flush enter 2024-03-25T18:31:28.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:28.016+0000 7fd077d15640 10 librados: watch_flush exit 2024-03-25T18:31:28.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:28.016+0000 7fd077d15640 20 client.4474.objecter shutdown clearing up homeless session... 2024-03-25T18:31:28.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:28.016+0000 7fd077d15640 10 client.4474.objecter successfully canceled tick 2024-03-25T18:31:28.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:28.016+0000 7fd077d15640 1 -- 172.21.15.3:0/2134261988 >> v1:172.21.15.3:6800/4195140966 conn(0x7fd05406de80 legacy=0x7fd054070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:28.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:28.016+0000 7fd077d15640 10 monclient: shutdown 2024-03-25T18:31:28.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:28.016+0000 7fd077d15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:28.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:28.016+0000 7fd077d15640 1 -- 172.21.15.3:0/2134261988 >> v1:172.21.15.16:6789/0 conn(0x7fd0700b4ff0 legacy=0x7fd0701db620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:28.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:28.016+0000 7fd077d15640 1 -- 172.21.15.3:0/2134261988 shutdown_connections 2024-03-25T18:31:28.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:28.016+0000 7fd077d15640 1 -- 172.21.15.3:0/2134261988 >> 172.21.15.3:0/2134261988 conn(0x7fd0700617a0 msgr2=0x7fd0700b2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:28.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:28.016+0000 7fd077d15640 1 -- 172.21.15.3:0/2134261988 shutdown_connections 2024-03-25T18:31:28.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:28.016+0000 7fd077d15640 1 -- 172.21.15.3:0/2134261988 wait complete. 2024-03-25T18:31:28.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:28.016+0000 7fd077d15640 1 librados: shutdown 2024-03-25T18:31:28.029 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:34.837 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:34.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.951+0000 7f0668bcb640 10 monclient: get_monmap_and_config 2024-03-25T18:31:34.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.951+0000 7f0668bcb640 10 monclient: build_initial_monmap 2024-03-25T18:31:34.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.951+0000 7f0668bcb640 10 monclient: monmap: 2024-03-25T18:31:34.952 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:31:34.953 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:34.953 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:31:34.952564+0000 2024-03-25T18:31:34.953 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:31:34.952564+0000 2024-03-25T18:31:34.953 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:34.953 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:34.953 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:31:34.953 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:31:34.953 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:31:34.953 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:34.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.952+0000 7f0668bcb640 1 Processor -- start 2024-03-25T18:31:34.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.952+0000 7f0668bcb640 1 -- start start 2024-03-25T18:31:34.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.952+0000 7f0668bcb640 10 monclient: init 2024-03-25T18:31:34.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.953+0000 7f0668bcb640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:34.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.953+0000 7f0668bcb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:31:34.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.953+0000 7f0668bcb640 10 monclient(hunting): picked mon.noname-c con 0x7f06640b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:31:34.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.953+0000 7f0668bcb640 10 monclient(hunting): picked mon.noname-b con 0x7f06640b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:31:34.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.953+0000 7f0668bcb640 10 monclient(hunting): picked mon.noname-a con 0x7f066413c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:31:34.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.953+0000 7f0668bcb640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06640ab0e0 con 0x7f066413c480 2024-03-25T18:31:34.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.953+0000 7f0668bcb640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06640ab3c0 con 0x7f06640b4ff0 2024-03-25T18:31:34.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.953+0000 7f0668bcb640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06640a9f30 con 0x7f06640b4670 2024-03-25T18:31:34.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.953+0000 7f0668bcb640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:34.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.953+0000 7f0668bcb640 10 monclient(hunting): authenticate will time out at 7749.213652s 2024-03-25T18:31:34.960 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.959+0000 7f066259c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f06640b4ff0 0x7f066413bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:38404/0 (socket says 172.21.15.3:38404) 2024-03-25T18:31:34.960 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.959+0000 7f066259c640 1 -- 172.21.15.3:0/2981637012 learned_addr learned my addr 172.21.15.3:0/2981637012 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:31:34.973 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.972+0000 7f0661d9b640 1 -- 172.21.15.3:0/2981637012 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386476762 0 0) 0x7f06640ab3c0 con 0x7f06640b4ff0 2024-03-25T18:31:34.973 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.972+0000 7f0661d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:31:34.973 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.972+0000 7f0661d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:34.974 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.972+0000 7f0661d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:34.974 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.972+0000 7f0661d9b640 10 monclient(hunting): my global_id is 4477 2024-03-25T18:31:34.974 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.972+0000 7f0661d9b640 1 -- 172.21.15.3:0/2981637012 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0650003520 con 0x7f06640b4ff0 2024-03-25T18:31:34.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.993+0000 7f0661d9b640 1 -- 172.21.15.3:0/2981637012 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 656008765 0 0) 0x7f06640a9f30 con 0x7f06640b4670 2024-03-25T18:31:34.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.993+0000 7f0661d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:31:34.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.993+0000 7f0661d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:34.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.993+0000 7f0661d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:34.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.993+0000 7f0661d9b640 10 monclient(hunting): my global_id is 4487 2024-03-25T18:31:34.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.993+0000 7f0661d9b640 1 -- 172.21.15.3:0/2981637012 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06640ab3c0 con 0x7f06640b4670 2024-03-25T18:31:34.996 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.995+0000 7f0661d9b640 1 -- 172.21.15.3:0/2981637012 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3771693329 0 0) 0x7f0650003520 con 0x7f06640b4ff0 2024-03-25T18:31:34.996 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.995+0000 7f0661d9b640 1 -- 172.21.15.3:0/2981637012 >> v1:172.21.15.3:6790/0 conn(0x7f06640b4670 legacy=0x7f06641425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:34.996 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.995+0000 7f0661d9b640 1 -- 172.21.15.3:0/2981637012 >> v1:172.21.15.3:6789/0 conn(0x7f066413c480 legacy=0x7f066414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:34.996 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.995+0000 7f0661d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:31:34.996 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.995+0000 7f0661d9b640 1 monclient: found mon.noname-c 2024-03-25T18:31:34.996 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.995+0000 7f0661d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:34.997 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.995+0000 7f0661d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:31:34.997 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.995+0000 7f0661d9b640 1 -- 172.21.15.3:0/2981637012 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06640a9b70 con 0x7f06640b4ff0 2024-03-25T18:31:34.997 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.995+0000 7f0661d9b640 10 monclient: _finish_auth 0 2024-03-25T18:31:34.997 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.995+0000 7f0661d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:31:34.997 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.995+0000 7f0661d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:34.997 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.995+0000 7f0661d9b640 1 -- 172.21.15.3:0/2981637012 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f0654001690 con 0x7f06640b4ff0 2024-03-25T18:31:34.997 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.995+0000 7f0661d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:34.997 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.995+0000 7f0661d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:34.997 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.995+0000 7f0661d9b640 10 monclient: dump: 2024-03-25T18:31:34.997 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:34.997 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:34.997 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:34.997 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:34.997 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:34.997 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:34.998 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:34.998 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:34.998 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:34.998 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:34.998 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.995+0000 7f0668bcb640 5 monclient: authenticate success, global_id 4477 2024-03-25T18:31:34.998 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:34.995+0000 7f0668bcb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:35.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.003+0000 7f0661d9b640 1 -- 172.21.15.3:0/2981637012 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06540017f0 con 0x7f06640b4ff0 2024-03-25T18:31:35.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.003+0000 7f0661d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:35.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.003+0000 7f0668bcb640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:35.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.003+0000 7f0668bcb640 4 set_mon_vals no callback set 2024-03-25T18:31:35.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.004+0000 7f0668bcb640 10 monclient: shutdown 2024-03-25T18:31:35.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.004+0000 7f0668bcb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:35.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.004+0000 7f0668bcb640 1 -- 172.21.15.3:0/2981637012 >> v1:172.21.15.16:6789/0 conn(0x7f06640b4ff0 legacy=0x7f066413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:35.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.004+0000 7f0668bcb640 1 -- 172.21.15.3:0/2981637012 shutdown_connections 2024-03-25T18:31:35.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.004+0000 7f0668bcb640 1 -- 172.21.15.3:0/2981637012 >> 172.21.15.3:0/2981637012 conn(0x7f06640617a0 msgr2=0x7f0664061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:35.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.004+0000 7f0668bcb640 1 -- 172.21.15.3:0/2981637012 shutdown_connections 2024-03-25T18:31:35.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.004+0000 7f0668bcb640 1 -- 172.21.15.3:0/2981637012 wait complete. 2024-03-25T18:31:35.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.005+0000 7f0668bcb640 10 monclient: build_initial_monmap 2024-03-25T18:31:35.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.005+0000 7f0668bcb640 10 monclient: monmap: 2024-03-25T18:31:35.007 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:31:35.007 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:35.007 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:31:35.007 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:31:35.007 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:35.007 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:31:35.008 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:31:35.008 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:31:35.008 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:31:35.008 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:35.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.006+0000 7f0668bcb640 1 librados: starting msgr at 2024-03-25T18:31:35.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.006+0000 7f0668bcb640 1 librados: starting objecter 2024-03-25T18:31:35.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.006+0000 7f0668bcb640 1 Processor -- start 2024-03-25T18:31:35.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.006+0000 7f0668bcb640 1 -- start start 2024-03-25T18:31:35.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.006+0000 7f0668bcb640 1 librados: setting wanted keys 2024-03-25T18:31:35.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.006+0000 7f0668bcb640 1 librados: calling monclient init 2024-03-25T18:31:35.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.006+0000 7f0668bcb640 10 monclient: init 2024-03-25T18:31:35.009 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.007+0000 7f0668bcb640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:35.009 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.007+0000 7f0668bcb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:31:35.009 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.007+0000 7f0668bcb640 10 monclient(hunting): picked mon.noname-b con 0x7f06640b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:31:35.009 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.007+0000 7f0668bcb640 10 monclient(hunting): picked mon.noname-c con 0x7f06640b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:31:35.009 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.007+0000 7f0668bcb640 10 monclient(hunting): picked mon.noname-a con 0x7f066413c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:31:35.010 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.007+0000 7f0668bcb640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06641445a0 con 0x7f066413c480 2024-03-25T18:31:35.010 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.007+0000 7f0668bcb640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0664144810 con 0x7f06640b4670 2024-03-25T18:31:35.010 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.007+0000 7f0668bcb640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0664144af0 con 0x7f06640b4ff0 2024-03-25T18:31:35.010 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.007+0000 7f0668bcb640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:35.010 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.007+0000 7f0668bcb640 10 monclient(hunting): authenticate will time out at 7749.267802s 2024-03-25T18:31:35.015 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.014+0000 7f0662d9d640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f066413c480 0x7f06641e2380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60942/0 (socket says 172.21.15.3:60942) 2024-03-25T18:31:35.015 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.014+0000 7f0662d9d640 1 -- 172.21.15.3:0/3981606891 learned_addr learned my addr 172.21.15.3:0/3981606891 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:31:35.017 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.016+0000 7f065affd640 10 client.?.objecter ms_handle_connect 0x7f066413c480 2024-03-25T18:31:35.017 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.016+0000 7f065affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:35.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.023+0000 7f065affd640 10 client.?.objecter ms_handle_connect 0x7f06640b4670 2024-03-25T18:31:35.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.023+0000 7f065affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:35.026 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.025+0000 7f065affd640 10 client.?.objecter ms_handle_connect 0x7f06640b4ff0 2024-03-25T18:31:35.026 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.025+0000 7f065affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:35.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.030+0000 7f065affd640 1 -- 172.21.15.3:0/3981606891 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2191570918 0 0) 0x7f0664144810 con 0x7f06640b4670 2024-03-25T18:31:35.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.030+0000 7f065affd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:35.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.030+0000 7f065affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:35.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.030+0000 7f065affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:35.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.030+0000 7f065affd640 10 monclient(hunting): my global_id is 4480 2024-03-25T18:31:35.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.030+0000 7f065affd640 1 -- 172.21.15.3:0/3981606891 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0644004eb0 con 0x7f06640b4670 2024-03-25T18:31:35.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.032+0000 7f065affd640 1 -- 172.21.15.3:0/3981606891 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3658489243 0 0) 0x7f0664144af0 con 0x7f06640b4ff0 2024-03-25T18:31:35.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.032+0000 7f065affd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:35.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.032+0000 7f065affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:35.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.032+0000 7f065affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:35.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.032+0000 7f065affd640 10 monclient(hunting): my global_id is 4490 2024-03-25T18:31:35.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.032+0000 7f065affd640 1 -- 172.21.15.3:0/3981606891 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0664144810 con 0x7f06640b4ff0 2024-03-25T18:31:35.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.032+0000 7f065affd640 1 -- 172.21.15.3:0/3981606891 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097503049 0 0) 0x7f06641445a0 con 0x7f066413c480 2024-03-25T18:31:35.034 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.032+0000 7f065affd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:35.034 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.032+0000 7f065affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:35.034 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.032+0000 7f065affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:35.034 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.032+0000 7f065affd640 10 monclient(hunting): my global_id is 4527 2024-03-25T18:31:35.034 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.032+0000 7f065affd640 1 -- 172.21.15.3:0/3981606891 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0664144af0 con 0x7f066413c480 2024-03-25T18:31:35.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.041+0000 7f065affd640 1 -- 172.21.15.3:0/3981606891 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4111631887 0 0) 0x7f0644004eb0 con 0x7f06640b4670 2024-03-25T18:31:35.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.041+0000 7f065affd640 1 -- 172.21.15.3:0/3981606891 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06641445a0 con 0x7f06640b4670 2024-03-25T18:31:35.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.041+0000 7f065affd640 1 -- 172.21.15.3:0/3981606891 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f0654001ca0 con 0x7f06640b4670 2024-03-25T18:31:35.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.041+0000 7f065affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:35.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.041+0000 7f065affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:35.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.041+0000 7f065affd640 10 monclient(hunting): dump: 2024-03-25T18:31:35.043 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:35.043 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:35.043 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:35.043 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:35.044 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:35.044 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:35.044 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:35.044 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:35.044 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:35.044 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:35.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.041+0000 7f065affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:35.060 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.059+0000 7f065affd640 1 -- 172.21.15.3:0/3981606891 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3445769602 0 0) 0x7f06641445a0 con 0x7f06640b4670 2024-03-25T18:31:35.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.059+0000 7f065affd640 1 -- 172.21.15.3:0/3981606891 >> v1:172.21.15.3:6790/0 conn(0x7f06640b4ff0 legacy=0x7f066413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:35.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.059+0000 7f065affd640 1 -- 172.21.15.3:0/3981606891 >> v1:172.21.15.3:6789/0 conn(0x7f066413c480 legacy=0x7f06641e2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:35.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.059+0000 7f065affd640 10 monclient: _finish_hunting 0 2024-03-25T18:31:35.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.059+0000 7f065affd640 1 monclient: found mon.b 2024-03-25T18:31:35.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.059+0000 7f065affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:35.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.059+0000 7f065affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:31:35.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.059+0000 7f065affd640 1 -- 172.21.15.3:0/3981606891 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0664144dd0 con 0x7f06640b4670 2024-03-25T18:31:35.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.059+0000 7f065affd640 10 monclient: _finish_auth 0 2024-03-25T18:31:35.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.059+0000 7f065affd640 10 monclient: _check_auth_tickets 2024-03-25T18:31:35.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.059+0000 7f065affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:35.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.059+0000 7f0668bcb640 5 monclient: authenticate success, global_id 4480 2024-03-25T18:31:35.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.059+0000 7f0668bcb640 10 monclient: _renew_subs 2024-03-25T18:31:35.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.059+0000 7f0668bcb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:31:35.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.059+0000 7f0668bcb640 1 -- 172.21.15.3:0/3981606891 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f066413f6d0 con 0x7f06640b4670 2024-03-25T18:31:35.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.059+0000 7f0668bcb640 10 client.4480.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:35.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.059+0000 7f0668bcb640 10 monclient: _renew_subs 2024-03-25T18:31:35.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.059+0000 7f0668bcb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:31:35.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.059+0000 7f0668bcb640 1 -- 172.21.15.3:0/3981606891 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0664143890 con 0x7f06640b4670 2024-03-25T18:31:35.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.059+0000 7f0668bcb640 1 librados: init done 2024-03-25T18:31:35.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.060+0000 7f0668bcb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:35.063 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.060+0000 7f0668bcb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:35.063 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.060+0000 7f0668bcb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:31:35.063 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.060+0000 7f0668bcb640 1 -- 172.21.15.3:0/3981606891 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f066413c1a0 con 0x7f06640b4670 2024-03-25T18:31:35.351 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.349+0000 7f065affd640 1 -- 172.21.15.3:0/3981606891 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06540011c0 con 0x7f06640b4670 2024-03-25T18:31:35.351 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.349+0000 7f065affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:35.351 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.349+0000 7f065affd640 1 -- 172.21.15.3:0/3981606891 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f0654004cb0 con 0x7f06640b4670 2024-03-25T18:31:35.351 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.349+0000 7f065affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:35.351 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.349+0000 7f065affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:35.351 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.349+0000 7f065affd640 10 monclient: dump: 2024-03-25T18:31:35.351 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:35.352 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:35.352 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:35.352 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:35.352 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:35.352 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:35.352 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:35.352 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:35.352 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:35.352 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:35.352 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.349+0000 7f066159a640 4 set_mon_vals no callback set 2024-03-25T18:31:35.353 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.351+0000 7f065affd640 1 -- 172.21.15.3:0/3981606891 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f065401ac80 con 0x7f06640b4670 2024-03-25T18:31:35.353 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.351+0000 7f065bfff640 10 monclient: get_auth_request con 0x7f064406ddd0 auth_method 0 2024-03-25T18:31:35.353 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.351+0000 7f065affd640 10 client.4480.objecter ms_handle_connect 0x7f064406ddd0 2024-03-25T18:31:35.357 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.356+0000 7f065affd640 1 -- 172.21.15.3:0/3981606891 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f0654054870 con 0x7f06640b4670 2024-03-25T18:31:35.357 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.356+0000 7f065affd640 10 client.4480.objecter ms_dispatch 0x7f0664000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:35.357 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.356+0000 7f065affd640 3 client.4480.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:35.357 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.356+0000 7f065affd640 3 client.4480.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:35.357 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.356+0000 7f065affd640 20 client.4480.objecter dump_active .. 0 homeless 2024-03-25T18:31:35.357 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.356+0000 7f065affd640 1 -- 172.21.15.3:0/3981606891 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f06540501b0 con 0x7f06640b4670 2024-03-25T18:31:35.357 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.356+0000 7f065affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:35.357 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.356+0000 7f065affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:35.621 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.620+0000 7f0668bcb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:35.621 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.620+0000 7f0668bcb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:35.621 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.620+0000 7f0668bcb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:31:35.621 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.620+0000 7f0668bcb640 1 -- 172.21.15.3:0/3981606891 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f066406dbd0 con 0x7f06640b4670 2024-03-25T18:31:35.644 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.643+0000 7f065affd640 1 -- 172.21.15.3:0/3981606891 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f06540522b0 con 0x7f06640b4670 2024-03-25T18:31:35.644 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.643+0000 7f065affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:35.644 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.643+0000 7f065affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:35.644 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:31:35.644 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:35.647 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.645+0000 7f0668bcb640 10 librados: watch_flush enter 2024-03-25T18:31:35.647 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.645+0000 7f0668bcb640 10 librados: watch_flush exit 2024-03-25T18:31:35.647 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.645+0000 7f0668bcb640 20 client.4480.objecter shutdown clearing up homeless session... 2024-03-25T18:31:35.647 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.645+0000 7f0668bcb640 10 client.4480.objecter successfully canceled tick 2024-03-25T18:31:35.647 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.645+0000 7f0668bcb640 1 -- 172.21.15.3:0/3981606891 >> v1:172.21.15.3:6800/4195140966 conn(0x7f064406ddd0 legacy=0x7f0644070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:35.647 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.645+0000 7f0668bcb640 10 monclient: shutdown 2024-03-25T18:31:35.647 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.645+0000 7f0668bcb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:35.647 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.645+0000 7f0668bcb640 1 -- 172.21.15.3:0/3981606891 >> v1:172.21.15.16:6789/0 conn(0x7f06640b4670 legacy=0x7f066413c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:35.647 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.645+0000 7f0668bcb640 1 -- 172.21.15.3:0/3981606891 shutdown_connections 2024-03-25T18:31:35.647 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.645+0000 7f0668bcb640 1 -- 172.21.15.3:0/3981606891 >> 172.21.15.3:0/3981606891 conn(0x7f06640617a0 msgr2=0x7f06640b70a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:35.647 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.645+0000 7f0668bcb640 1 -- 172.21.15.3:0/3981606891 shutdown_connections 2024-03-25T18:31:35.647 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.645+0000 7f0668bcb640 1 -- 172.21.15.3:0/3981606891 wait complete. 2024-03-25T18:31:35.647 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:35.645+0000 7f0668bcb640 1 librados: shutdown 2024-03-25T18:31:35.658 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:42.465 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:42.538 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:06:56.267 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:42.581 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.580+0000 7f34fcce6640 10 monclient: get_monmap_and_config 2024-03-25T18:31:42.582 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.580+0000 7f34fcce6640 10 monclient: build_initial_monmap 2024-03-25T18:31:42.582 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.580+0000 7f34fcce6640 10 monclient: monmap: 2024-03-25T18:31:42.582 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:31:42.582 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:42.582 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:31:42.581657+0000 2024-03-25T18:31:42.582 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:31:42.581657+0000 2024-03-25T18:31:42.582 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:42.582 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:42.582 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:31:42.582 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:31:42.582 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:31:42.582 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:42.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.581+0000 7f34fcce6640 1 Processor -- start 2024-03-25T18:31:42.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.581+0000 7f34fcce6640 1 -- start start 2024-03-25T18:31:42.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.581+0000 7f34fcce6640 10 monclient: init 2024-03-25T18:31:42.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.581+0000 7f34fcce6640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:42.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.581+0000 7f34fcce6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:31:42.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.581+0000 7f34fcce6640 10 monclient(hunting): picked mon.noname-a con 0x7f34f80b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:31:42.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.581+0000 7f34fcce6640 10 monclient(hunting): picked mon.noname-c con 0x7f34f80b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:31:42.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.581+0000 7f34fcce6640 10 monclient(hunting): picked mon.noname-b con 0x7f34f813c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:31:42.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.581+0000 7f34fcce6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f80ab0e0 con 0x7f34f80b4ff0 2024-03-25T18:31:42.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.581+0000 7f34fcce6640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f80ab3c0 con 0x7f34f80b4670 2024-03-25T18:31:42.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.581+0000 7f34fcce6640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f80a9f30 con 0x7f34f813c480 2024-03-25T18:31:42.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.581+0000 7f34fcce6640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:42.585 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.581+0000 7f34fcce6640 10 monclient(hunting): authenticate will time out at 7756.842185s 2024-03-25T18:31:42.585 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.582+0000 7f34edd9b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f34f80b4670 0x7f34f81425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:46102/0 (socket says 172.21.15.3:46102) 2024-03-25T18:31:42.585 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.582+0000 7f34edd9b640 1 -- 172.21.15.3:0/3370708320 learned_addr learned my addr 172.21.15.3:0/3370708320 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:31:42.604 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:06:56.333 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:42.628 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:06:56.357 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:42.671 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:06:56.400 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:42.672 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:06:56.364 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:42.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.676+0000 7f34f5d9b640 1 -- 172.21.15.3:0/3370708320 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1899231644 0 0) 0x7f34f80ab3c0 con 0x7f34f80b4670 2024-03-25T18:31:42.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.676+0000 7f34f5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:31:42.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.676+0000 7f34f5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:42.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.676+0000 7f34f5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:42.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.676+0000 7f34f5d9b640 10 monclient(hunting): my global_id is 4483 2024-03-25T18:31:42.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.676+0000 7f34f5d9b640 1 -- 172.21.15.3:0/3370708320 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34e4003520 con 0x7f34f80b4670 2024-03-25T18:31:42.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.679+0000 7f34f5d9b640 1 -- 172.21.15.3:0/3370708320 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3593852168 0 0) 0x7f34f80a9f30 con 0x7f34f813c480 2024-03-25T18:31:42.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.679+0000 7f34f5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:31:42.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.679+0000 7f34f5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:42.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.679+0000 7f34f5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:42.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.679+0000 7f34f5d9b640 10 monclient(hunting): my global_id is 4493 2024-03-25T18:31:42.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.679+0000 7f34f5d9b640 1 -- 172.21.15.3:0/3370708320 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34f80ab3c0 con 0x7f34f813c480 2024-03-25T18:31:42.682 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:06:56.411 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:42.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.685+0000 7f34f5d9b640 1 -- 172.21.15.3:0/3370708320 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4083360463 0 0) 0x7f34e4003520 con 0x7f34f80b4670 2024-03-25T18:31:42.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.685+0000 7f34f5d9b640 1 -- 172.21.15.3:0/3370708320 >> v1:172.21.15.3:6790/0 conn(0x7f34f813c480 legacy=0x7f34f814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:42.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.685+0000 7f34f5d9b640 1 -- 172.21.15.3:0/3370708320 >> v1:172.21.15.3:6789/0 conn(0x7f34f80b4ff0 legacy=0x7f34f813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:42.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.685+0000 7f34f5d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:31:42.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.685+0000 7f34f5d9b640 1 monclient: found mon.noname-c 2024-03-25T18:31:42.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.685+0000 7f34f5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:42.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.685+0000 7f34f5d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:31:42.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.685+0000 7f34f5d9b640 1 -- 172.21.15.3:0/3370708320 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f80a9b70 con 0x7f34f80b4670 2024-03-25T18:31:42.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.685+0000 7f34f5d9b640 10 monclient: _finish_auth 0 2024-03-25T18:31:42.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.685+0000 7f34f5d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:31:42.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.685+0000 7f34f5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:42.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.685+0000 7f34f5d9b640 1 -- 172.21.15.3:0/3370708320 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f34e0001690 con 0x7f34f80b4670 2024-03-25T18:31:42.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.685+0000 7f34fcce6640 5 monclient: authenticate success, global_id 4483 2024-03-25T18:31:42.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.685+0000 7f34fcce6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:42.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.685+0000 7f34f5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:42.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.685+0000 7f34f5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:42.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.685+0000 7f34f5d9b640 10 monclient: dump: 2024-03-25T18:31:42.688 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:42.688 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:42.688 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:42.688 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:42.688 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:42.688 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:42.688 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:42.689 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:42.689 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:42.689 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:42.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.685+0000 7f34fcce6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:42.689 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:06:56.379 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:42.707 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.706+0000 7f34f5d9b640 1 -- 172.21.15.3:0/3370708320 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34e00017f0 con 0x7f34f80b4670 2024-03-25T18:31:42.707 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.706+0000 7f34f5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:42.707 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.706+0000 7f34f5d9b640 1 -- 172.21.15.3:0/3370708320 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f34e0001c00 con 0x7f34f80b4670 2024-03-25T18:31:42.707 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.706+0000 7f34fcce6640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:42.707 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.706+0000 7f34fcce6640 4 set_mon_vals no callback set 2024-03-25T18:31:42.707 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.706+0000 7f34f5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:42.707 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.706+0000 7f34f5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:42.707 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.706+0000 7f34f5d9b640 10 monclient: dump: 2024-03-25T18:31:42.707 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:42.708 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:42.708 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:42.708 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:42.708 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:42.708 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:42.708 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:42.708 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:42.708 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:42.708 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:42.709 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.706+0000 7f34fcce6640 10 monclient: shutdown 2024-03-25T18:31:42.709 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.706+0000 7f34fcce6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:42.709 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.706+0000 7f34fcce6640 1 -- 172.21.15.3:0/3370708320 >> v1:172.21.15.16:6789/0 conn(0x7f34f80b4670 legacy=0x7f34f81425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:42.709 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.706+0000 7f34fcce6640 1 -- 172.21.15.3:0/3370708320 shutdown_connections 2024-03-25T18:31:42.710 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.706+0000 7f34fcce6640 1 -- 172.21.15.3:0/3370708320 >> 172.21.15.3:0/3370708320 conn(0x7f34f80617a0 msgr2=0x7f34f8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:42.710 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.706+0000 7f34fcce6640 1 -- 172.21.15.3:0/3370708320 shutdown_connections 2024-03-25T18:31:42.710 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.706+0000 7f34fcce6640 1 -- 172.21.15.3:0/3370708320 wait complete. 2024-03-25T18:31:42.710 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.707+0000 7f34fcce6640 10 monclient: build_initial_monmap 2024-03-25T18:31:42.710 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.707+0000 7f34fcce6640 10 monclient: monmap: 2024-03-25T18:31:42.710 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:31:42.710 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:42.710 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:31:42.710 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:31:42.710 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:42.710 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:31:42.710 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:31:42.710 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:31:42.711 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:31:42.711 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:42.711 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.708+0000 7f34fcce6640 1 librados: starting msgr at 2024-03-25T18:31:42.711 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.708+0000 7f34fcce6640 1 librados: starting objecter 2024-03-25T18:31:42.711 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.708+0000 7f34fcce6640 1 Processor -- start 2024-03-25T18:31:42.711 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.708+0000 7f34fcce6640 1 -- start start 2024-03-25T18:31:42.711 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.708+0000 7f34fcce6640 1 librados: setting wanted keys 2024-03-25T18:31:42.711 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.708+0000 7f34fcce6640 1 librados: calling monclient init 2024-03-25T18:31:42.711 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.708+0000 7f34fcce6640 10 monclient: init 2024-03-25T18:31:42.712 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.709+0000 7f34fcce6640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:42.712 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.709+0000 7f34fcce6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:31:42.712 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.709+0000 7f34fcce6640 10 monclient(hunting): picked mon.noname-a con 0x7f34f80b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:31:42.712 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.709+0000 7f34fcce6640 10 monclient(hunting): picked mon.noname-b con 0x7f34f80b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:31:42.712 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.709+0000 7f34fcce6640 10 monclient(hunting): picked mon.noname-c con 0x7f34f813c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:31:42.712 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.709+0000 7f34fcce6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f8144a50 con 0x7f34f80b4670 2024-03-25T18:31:42.712 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.709+0000 7f34fcce6640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f813ec30 con 0x7f34f80b4ff0 2024-03-25T18:31:42.712 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.709+0000 7f34fcce6640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f813edc0 con 0x7f34f813c480 2024-03-25T18:31:42.712 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.709+0000 7f34fcce6640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:42.712 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.709+0000 7f34fcce6640 10 monclient(hunting): authenticate will time out at 7756.969628s 2024-03-25T18:31:42.713 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:06:56.405 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:42.717 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:06:56.447 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:42.733 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.732+0000 7f34f659c640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f34f80b4670 0x7f34f81457a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:36936/0 (socket says 172.21.15.3:36936) 2024-03-25T18:31:42.733 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.732+0000 7f34f659c640 1 -- 172.21.15.3:0/3394322643 learned_addr learned my addr 172.21.15.3:0/3394322643 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:31:42.741 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.740+0000 7f34ef7fe640 10 client.?.objecter ms_handle_connect 0x7f34f80b4670 2024-03-25T18:31:42.741 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.740+0000 7f34ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:42.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.763+0000 7f34ef7fe640 10 client.?.objecter ms_handle_connect 0x7f34f813c480 2024-03-25T18:31:42.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.763+0000 7f34ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:42.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.772+0000 7f34ef7fe640 1 -- 172.21.15.3:0/3394322643 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660877797 0 0) 0x7f34f8144a50 con 0x7f34f80b4670 2024-03-25T18:31:42.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.772+0000 7f34ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:42.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.772+0000 7f34ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:42.774 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.772+0000 7f34ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:42.774 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.772+0000 7f34ef7fe640 10 monclient(hunting): my global_id is 4533 2024-03-25T18:31:42.774 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.772+0000 7f34ef7fe640 1 -- 172.21.15.3:0/3394322643 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34d8004eb0 con 0x7f34f80b4670 2024-03-25T18:31:42.799 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.797+0000 7f34ef7fe640 1 -- 172.21.15.3:0/3394322643 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1839141863 0 0) 0x7f34d8004eb0 con 0x7f34f80b4670 2024-03-25T18:31:42.799 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.797+0000 7f34ef7fe640 1 -- 172.21.15.3:0/3394322643 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34f8144a50 con 0x7f34f80b4670 2024-03-25T18:31:42.799 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.797+0000 7f34ef7fe640 1 -- 172.21.15.3:0/3394322643 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f34e80019d0 con 0x7f34f80b4670 2024-03-25T18:31:42.799 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.797+0000 7f34ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:42.799 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.797+0000 7f34ef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:42.799 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.797+0000 7f34ef7fe640 10 monclient(hunting): dump: 2024-03-25T18:31:42.799 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:42.799 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:42.799 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:42.799 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:42.799 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:42.799 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:42.800 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:42.800 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:42.800 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:42.800 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:42.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.797+0000 7f34ef7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:42.801 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.800+0000 7f34ef7fe640 10 client.?.objecter ms_handle_connect 0x7f34f80b4ff0 2024-03-25T18:31:42.801 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.800+0000 7f34ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:42.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.817+0000 7f34ef7fe640 1 -- 172.21.15.3:0/3394322643 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4110458950 0 0) 0x7f34f8144a50 con 0x7f34f80b4670 2024-03-25T18:31:42.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.817+0000 7f34ef7fe640 1 -- 172.21.15.3:0/3394322643 >> v1:172.21.15.3:6790/0 conn(0x7f34f813c480 legacy=0x7f34f81e26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:42.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.817+0000 7f34ef7fe640 1 -- 172.21.15.3:0/3394322643 >> v1:172.21.15.16:6789/0 conn(0x7f34f80b4ff0 legacy=0x7f34f8145eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:42.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.817+0000 7f34ef7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:31:42.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.817+0000 7f34ef7fe640 1 monclient: found mon.a 2024-03-25T18:31:42.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.817+0000 7f34ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:42.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.817+0000 7f34ef7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:31:42.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.817+0000 7f34ef7fe640 1 -- 172.21.15.3:0/3394322643 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f813f6e0 con 0x7f34f80b4670 2024-03-25T18:31:42.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.817+0000 7f34ef7fe640 10 monclient: _finish_auth 0 2024-03-25T18:31:42.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.817+0000 7f34ef7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:31:42.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.817+0000 7f34ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:42.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.817+0000 7f34fcce6640 5 monclient: authenticate success, global_id 4533 2024-03-25T18:31:42.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.817+0000 7f34fcce6640 10 monclient: _renew_subs 2024-03-25T18:31:42.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.817+0000 7f34fcce6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:31:42.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.817+0000 7f34fcce6640 1 -- 172.21.15.3:0/3394322643 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f813f930 con 0x7f34f80b4670 2024-03-25T18:31:42.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.817+0000 7f34fcce6640 10 client.4533.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:42.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.817+0000 7f34fcce6640 10 monclient: _renew_subs 2024-03-25T18:31:42.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.817+0000 7f34fcce6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:31:42.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.817+0000 7f34fcce6640 1 -- 172.21.15.3:0/3394322643 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f8143ab0 con 0x7f34f80b4670 2024-03-25T18:31:42.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.817+0000 7f34fcce6640 1 librados: init done 2024-03-25T18:31:42.821 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.818+0000 7f34fcce6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:42.821 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.818+0000 7f34fcce6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:42.821 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.818+0000 7f34fcce6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:31:42.821 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:42.818+0000 7f34fcce6640 1 -- 172.21.15.3:0/3394322643 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34f8145150 con 0x7f34f80b4670 2024-03-25T18:31:43.097 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.096+0000 7f34ef7fe640 1 -- 172.21.15.3:0/3394322643 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34e8001c90 con 0x7f34f80b4670 2024-03-25T18:31:43.098 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.096+0000 7f34ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:43.098 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.096+0000 7f34ef7fe640 1 -- 172.21.15.3:0/3394322643 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f34e80038c0 con 0x7f34f80b4670 2024-03-25T18:31:43.098 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.096+0000 7f34ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:43.098 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.096+0000 7f34ef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:43.098 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.096+0000 7f34ef7fe640 10 monclient: dump: 2024-03-25T18:31:43.098 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:43.098 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:43.098 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:43.098 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:43.098 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:43.099 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:43.099 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:43.099 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:43.099 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:43.099 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:43.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.096+0000 7f34f559a640 4 set_mon_vals no callback set 2024-03-25T18:31:43.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.097+0000 7f34ef7fe640 1 -- 172.21.15.3:0/3394322643 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f34e8003ae0 con 0x7f34f80b4670 2024-03-25T18:31:43.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.098+0000 7f34edd9b640 10 monclient: get_auth_request con 0x7f34d806dc00 auth_method 0 2024-03-25T18:31:43.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.098+0000 7f34ef7fe640 10 client.4533.objecter ms_handle_connect 0x7f34d806dc00 2024-03-25T18:31:43.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.102+0000 7f34ef7fe640 1 -- 172.21.15.3:0/3394322643 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f34e8054690 con 0x7f34f80b4670 2024-03-25T18:31:43.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.102+0000 7f34ef7fe640 10 client.4533.objecter ms_dispatch 0x7f34f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:43.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.102+0000 7f34ef7fe640 3 client.4533.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:43.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.102+0000 7f34ef7fe640 3 client.4533.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:43.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.103+0000 7f34ef7fe640 20 client.4533.objecter dump_active .. 0 homeless 2024-03-25T18:31:43.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.103+0000 7f34ef7fe640 1 -- 172.21.15.3:0/3394322643 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f34e80501a0 con 0x7f34f80b4670 2024-03-25T18:31:43.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.103+0000 7f34ef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:43.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.103+0000 7f34ef7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:43.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.344+0000 7f34fcce6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:43.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.344+0000 7f34fcce6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:43.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.344+0000 7f34fcce6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:31:43.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.344+0000 7f34fcce6640 1 -- 172.21.15.3:0/3394322643 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34f806dbd0 con 0x7f34f80b4670 2024-03-25T18:31:43.381 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.379+0000 7f34ef7fe640 1 -- 172.21.15.3:0/3394322643 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f34e80520d0 con 0x7f34f80b4670 2024-03-25T18:31:43.381 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.379+0000 7f34ef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:43.381 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.379+0000 7f34ef7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:43.381 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:31:43.381 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:43.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.382+0000 7f34fcce6640 10 librados: watch_flush enter 2024-03-25T18:31:43.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.382+0000 7f34fcce6640 10 librados: watch_flush exit 2024-03-25T18:31:43.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.382+0000 7f34fcce6640 20 client.4533.objecter shutdown clearing up homeless session... 2024-03-25T18:31:43.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.382+0000 7f34fcce6640 10 client.4533.objecter successfully canceled tick 2024-03-25T18:31:43.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.382+0000 7f34fcce6640 1 -- 172.21.15.3:0/3394322643 >> v1:172.21.15.3:6800/4195140966 conn(0x7f34d806dc00 legacy=0x7f34d80700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:43.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.382+0000 7f34fcce6640 10 monclient: shutdown 2024-03-25T18:31:43.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.382+0000 7f34fcce6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:43.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.382+0000 7f34fcce6640 1 -- 172.21.15.3:0/3394322643 >> v1:172.21.15.3:6789/0 conn(0x7f34f80b4670 legacy=0x7f34f81457a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:43.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.382+0000 7f34fcce6640 1 -- 172.21.15.3:0/3394322643 shutdown_connections 2024-03-25T18:31:43.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.382+0000 7f34fcce6640 1 -- 172.21.15.3:0/3394322643 >> 172.21.15.3:0/3394322643 conn(0x7f34f80617a0 msgr2=0x7f34f81407f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:43.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.382+0000 7f34fcce6640 1 -- 172.21.15.3:0/3394322643 shutdown_connections 2024-03-25T18:31:43.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.382+0000 7f34fcce6640 1 -- 172.21.15.3:0/3394322643 wait complete. 2024-03-25T18:31:43.385 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:43.382+0000 7f34fcce6640 1 librados: shutdown 2024-03-25T18:31:43.395 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:50.201 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:50.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.317+0000 7fe609f15640 10 monclient: get_monmap_and_config 2024-03-25T18:31:50.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.317+0000 7fe609f15640 10 monclient: build_initial_monmap 2024-03-25T18:31:50.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.317+0000 7fe609f15640 10 monclient: monmap: 2024-03-25T18:31:50.319 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:31:50.319 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:50.319 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:31:50.318672+0000 2024-03-25T18:31:50.319 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:31:50.318672+0000 2024-03-25T18:31:50.319 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:50.319 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:50.319 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:31:50.319 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:31:50.319 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:31:50.319 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:50.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.318+0000 7fe609f15640 1 Processor -- start 2024-03-25T18:31:50.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.318+0000 7fe609f15640 1 -- start start 2024-03-25T18:31:50.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.318+0000 7fe609f15640 10 monclient: init 2024-03-25T18:31:50.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.318+0000 7fe609f15640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:50.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.318+0000 7fe609f15640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:31:50.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.319+0000 7fe609f15640 10 monclient(hunting): picked mon.noname-a con 0x7fe6040b4e70 addr v1:172.21.15.3:6789/0 2024-03-25T18:31:50.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.319+0000 7fe609f15640 10 monclient(hunting): picked mon.noname-b con 0x7fe6040b44b0 addr v1:172.21.15.3:6790/0 2024-03-25T18:31:50.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.319+0000 7fe609f15640 10 monclient(hunting): picked mon.noname-c con 0x7fe6040b1b00 addr v1:172.21.15.16:6789/0 2024-03-25T18:31:50.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.319+0000 7fe609f15640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6040a96f0 con 0x7fe6040b4e70 2024-03-25T18:31:50.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.319+0000 7fe609f15640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6040ab0e0 con 0x7fe6040b1b00 2024-03-25T18:31:50.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.319+0000 7fe609f15640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe604145f70 con 0x7fe6040b44b0 2024-03-25T18:31:50.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.319+0000 7fe609f15640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:50.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.319+0000 7fe609f15640 10 monclient(hunting): authenticate will time out at 7764.579425s 2024-03-25T18:31:50.326 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.325+0000 7fe603fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe6040b1b00 0x7fe60414dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:42490/0 (socket says 172.21.15.3:42490) 2024-03-25T18:31:50.326 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.325+0000 7fe603fff640 1 -- 172.21.15.3:0/3454545062 learned_addr learned my addr 172.21.15.3:0/3454545062 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:31:50.337 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.336+0000 7fe6027fc640 1 -- 172.21.15.3:0/3454545062 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821485990 0 0) 0x7fe6040a96f0 con 0x7fe6040b4e70 2024-03-25T18:31:50.337 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.336+0000 7fe6027fc640 10 monclient(hunting): _negotiate 2024-03-25T18:31:50.337 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.336+0000 7fe6027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:50.337 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.336+0000 7fe6027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:50.337 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.336+0000 7fe6027fc640 10 monclient(hunting): my global_id is 4536 2024-03-25T18:31:50.337 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.336+0000 7fe6027fc640 1 -- 172.21.15.3:0/3454545062 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5ec003520 con 0x7fe6040b4e70 2024-03-25T18:31:50.339 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.338+0000 7fe6027fc640 1 -- 172.21.15.3:0/3454545062 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239950410 0 0) 0x7fe6040ab0e0 con 0x7fe6040b1b00 2024-03-25T18:31:50.339 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.338+0000 7fe6027fc640 10 monclient(hunting): _negotiate 2024-03-25T18:31:50.340 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.338+0000 7fe6027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:50.340 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.338+0000 7fe6027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:50.340 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.338+0000 7fe6027fc640 10 monclient(hunting): my global_id is 4489 2024-03-25T18:31:50.340 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.338+0000 7fe6027fc640 1 -- 172.21.15.3:0/3454545062 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6040a96f0 con 0x7fe6040b1b00 2024-03-25T18:31:50.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.343+0000 7fe6027fc640 1 -- 172.21.15.3:0/3454545062 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4257545626 0 0) 0x7fe5ec003520 con 0x7fe6040b4e70 2024-03-25T18:31:50.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.343+0000 7fe6027fc640 1 -- 172.21.15.3:0/3454545062 >> v1:172.21.15.3:6790/0 conn(0x7fe6040b44b0 legacy=0x7fe604145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:31:50.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.343+0000 7fe6027fc640 1 -- 172.21.15.3:0/3454545062 >> v1:172.21.15.16:6789/0 conn(0x7fe6040b1b00 legacy=0x7fe60414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:50.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.343+0000 7fe6027fc640 10 monclient: _finish_hunting 0 2024-03-25T18:31:50.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.343+0000 7fe6027fc640 1 monclient: found mon.noname-a 2024-03-25T18:31:50.346 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.344+0000 7fe6027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:50.346 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.344+0000 7fe6027fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:31:50.346 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.344+0000 7fe6027fc640 1 -- 172.21.15.3:0/3454545062 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6040a9b80 con 0x7fe6040b4e70 2024-03-25T18:31:50.346 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.344+0000 7fe6027fc640 10 monclient: _finish_auth 0 2024-03-25T18:31:50.346 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.344+0000 7fe6027fc640 10 monclient: _check_auth_tickets 2024-03-25T18:31:50.346 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.344+0000 7fe6027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:50.346 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.344+0000 7fe6027fc640 1 -- 172.21.15.3:0/3454545062 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fe5f4001690 con 0x7fe6040b4e70 2024-03-25T18:31:50.346 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.344+0000 7fe6027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:50.346 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.344+0000 7fe6027fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:50.346 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.344+0000 7fe6027fc640 10 monclient: dump: 2024-03-25T18:31:50.346 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:50.346 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:50.346 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:50.346 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:50.346 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:50.347 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:50.347 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:50.347 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:50.347 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:50.347 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:50.347 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.344+0000 7fe609f15640 5 monclient: authenticate success, global_id 4536 2024-03-25T18:31:50.347 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.344+0000 7fe609f15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:50.347 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.346+0000 7fe6027fc640 1 -- 172.21.15.3:0/3454545062 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5f40017f0 con 0x7fe6040b4e70 2024-03-25T18:31:50.347 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.346+0000 7fe6027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:50.347 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.346+0000 7fe609f15640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:50.347 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.346+0000 7fe609f15640 4 set_mon_vals no callback set 2024-03-25T18:31:50.349 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.347+0000 7fe609f15640 10 monclient: shutdown 2024-03-25T18:31:50.349 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.347+0000 7fe609f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:50.349 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.347+0000 7fe609f15640 1 -- 172.21.15.3:0/3454545062 >> v1:172.21.15.3:6789/0 conn(0x7fe6040b4e70 legacy=0x7fe60413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:50.349 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.347+0000 7fe609f15640 1 -- 172.21.15.3:0/3454545062 shutdown_connections 2024-03-25T18:31:50.349 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.347+0000 7fe609f15640 1 -- 172.21.15.3:0/3454545062 >> 172.21.15.3:0/3454545062 conn(0x7fe6040617a0 msgr2=0x7fe604061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:50.349 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.347+0000 7fe609f15640 1 -- 172.21.15.3:0/3454545062 shutdown_connections 2024-03-25T18:31:50.349 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.348+0000 7fe609f15640 1 -- 172.21.15.3:0/3454545062 wait complete. 2024-03-25T18:31:50.350 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.348+0000 7fe609f15640 10 monclient: build_initial_monmap 2024-03-25T18:31:50.350 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.348+0000 7fe609f15640 10 monclient: monmap: 2024-03-25T18:31:50.350 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:31:50.350 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:50.350 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:31:50.350 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:31:50.350 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:50.350 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:31:50.350 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:31:50.350 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:31:50.350 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:31:50.350 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:50.351 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.349+0000 7fe609f15640 1 librados: starting msgr at 2024-03-25T18:31:50.351 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.349+0000 7fe609f15640 1 librados: starting objecter 2024-03-25T18:31:50.351 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.349+0000 7fe609f15640 1 Processor -- start 2024-03-25T18:31:50.352 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.349+0000 7fe609f15640 1 -- start start 2024-03-25T18:31:50.352 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.349+0000 7fe609f15640 1 librados: setting wanted keys 2024-03-25T18:31:50.352 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.349+0000 7fe609f15640 1 librados: calling monclient init 2024-03-25T18:31:50.352 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.349+0000 7fe609f15640 10 monclient: init 2024-03-25T18:31:50.352 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.350+0000 7fe609f15640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:50.352 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.350+0000 7fe609f15640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:31:50.352 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.350+0000 7fe609f15640 10 monclient(hunting): picked mon.noname-b con 0x7fe6040b1b00 addr v1:172.21.15.16:6789/0 2024-03-25T18:31:50.352 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.350+0000 7fe609f15640 10 monclient(hunting): picked mon.noname-c con 0x7fe6040b44b0 addr v1:172.21.15.3:6790/0 2024-03-25T18:31:50.352 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.350+0000 7fe609f15640 10 monclient(hunting): picked mon.noname-a con 0x7fe6040b4e70 addr v1:172.21.15.3:6789/0 2024-03-25T18:31:50.352 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.350+0000 7fe609f15640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe604137a50 con 0x7fe6040b4e70 2024-03-25T18:31:50.352 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.350+0000 7fe609f15640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe604134de0 con 0x7fe6040b1b00 2024-03-25T18:31:50.352 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.350+0000 7fe609f15640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe604134f70 con 0x7fe6040b44b0 2024-03-25T18:31:50.353 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.350+0000 7fe609f15640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:50.353 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.350+0000 7fe609f15640 10 monclient(hunting): authenticate will time out at 7764.610801s 2024-03-25T18:31:50.366 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.365+0000 7fe603fff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fe6040b4e70 0x7fe604141f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57950/0 (socket says 172.21.15.3:57950) 2024-03-25T18:31:50.366 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.365+0000 7fe603fff640 1 -- 172.21.15.3:0/2739526003 learned_addr learned my addr 172.21.15.3:0/2739526003 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:31:50.367 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.366+0000 7fe5ebfff640 10 client.?.objecter ms_handle_connect 0x7fe6040b4e70 2024-03-25T18:31:50.367 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.366+0000 7fe5ebfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:50.373 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.372+0000 7fe5ebfff640 10 client.?.objecter ms_handle_connect 0x7fe6040b44b0 2024-03-25T18:31:50.374 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.372+0000 7fe5ebfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:50.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.376+0000 7fe5ebfff640 10 client.?.objecter ms_handle_connect 0x7fe6040b1b00 2024-03-25T18:31:50.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.376+0000 7fe5ebfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:50.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.381+0000 7fe5ebfff640 1 -- 172.21.15.3:0/2739526003 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3571006046 0 0) 0x7fe604137a50 con 0x7fe6040b4e70 2024-03-25T18:31:50.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.381+0000 7fe5ebfff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:50.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.381+0000 7fe5ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:50.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.381+0000 7fe5ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:50.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.381+0000 7fe5ebfff640 10 monclient(hunting): my global_id is 4539 2024-03-25T18:31:50.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.381+0000 7fe5ebfff640 1 -- 172.21.15.3:0/2739526003 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5f0004eb0 con 0x7fe6040b4e70 2024-03-25T18:31:50.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.385+0000 7fe5ebfff640 1 -- 172.21.15.3:0/2739526003 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528847193 0 0) 0x7fe604134de0 con 0x7fe6040b1b00 2024-03-25T18:31:50.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.385+0000 7fe5ebfff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:50.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.385+0000 7fe5ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:50.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.385+0000 7fe5ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:50.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.385+0000 7fe5ebfff640 10 monclient(hunting): my global_id is 4492 2024-03-25T18:31:50.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.385+0000 7fe5ebfff640 1 -- 172.21.15.3:0/2739526003 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe604137a50 con 0x7fe6040b1b00 2024-03-25T18:31:50.393 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.392+0000 7fe5ebfff640 1 -- 172.21.15.3:0/2739526003 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 825597951 0 0) 0x7fe5f0004eb0 con 0x7fe6040b4e70 2024-03-25T18:31:50.393 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.392+0000 7fe5ebfff640 1 -- 172.21.15.3:0/2739526003 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe604134de0 con 0x7fe6040b4e70 2024-03-25T18:31:50.393 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.392+0000 7fe5ebfff640 1 -- 172.21.15.3:0/2739526003 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fe5fc001cb0 con 0x7fe6040b4e70 2024-03-25T18:31:50.393 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.392+0000 7fe5ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:50.393 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.392+0000 7fe5ebfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:50.394 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.392+0000 7fe5ebfff640 10 monclient(hunting): dump: 2024-03-25T18:31:50.394 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:50.394 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:50.394 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:50.394 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:50.394 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:50.394 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:50.394 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:50.394 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:50.394 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:50.394 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:50.394 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.392+0000 7fe5ebfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:50.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.394+0000 7fe5ebfff640 1 -- 172.21.15.3:0/2739526003 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2275324971 0 0) 0x7fe604134f70 con 0x7fe6040b44b0 2024-03-25T18:31:50.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.394+0000 7fe5ebfff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:50.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.394+0000 7fe5ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:50.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.394+0000 7fe5ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:50.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.394+0000 7fe5ebfff640 10 monclient(hunting): my global_id is 4496 2024-03-25T18:31:50.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.394+0000 7fe5ebfff640 1 -- 172.21.15.3:0/2739526003 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5f0004eb0 con 0x7fe6040b44b0 2024-03-25T18:31:50.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.404+0000 7fe5ebfff640 1 -- 172.21.15.3:0/2739526003 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2245299681 0 0) 0x7fe604134de0 con 0x7fe6040b4e70 2024-03-25T18:31:50.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.404+0000 7fe5ebfff640 1 -- 172.21.15.3:0/2739526003 >> v1:172.21.15.3:6790/0 conn(0x7fe6040b44b0 legacy=0x7fe604141820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:50.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.404+0000 7fe5ebfff640 1 -- 172.21.15.3:0/2739526003 >> v1:172.21.15.16:6789/0 conn(0x7fe6040b1b00 legacy=0x7fe6041dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:50.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.404+0000 7fe5ebfff640 10 monclient: _finish_hunting 0 2024-03-25T18:31:50.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.404+0000 7fe5ebfff640 1 monclient: found mon.a 2024-03-25T18:31:50.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.404+0000 7fe5ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:50.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.404+0000 7fe5ebfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:31:50.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.404+0000 7fe5ebfff640 1 -- 172.21.15.3:0/2739526003 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6041db2a0 con 0x7fe6040b4e70 2024-03-25T18:31:50.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.404+0000 7fe5ebfff640 10 monclient: _finish_auth 0 2024-03-25T18:31:50.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.404+0000 7fe5ebfff640 10 monclient: _check_auth_tickets 2024-03-25T18:31:50.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.404+0000 7fe5ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:50.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.404+0000 7fe609f15640 5 monclient: authenticate success, global_id 4539 2024-03-25T18:31:50.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.404+0000 7fe609f15640 10 monclient: _renew_subs 2024-03-25T18:31:50.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.404+0000 7fe609f15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:31:50.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.404+0000 7fe609f15640 1 -- 172.21.15.3:0/2739526003 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6041db550 con 0x7fe6040b4e70 2024-03-25T18:31:50.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.404+0000 7fe609f15640 10 client.4539.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:50.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.404+0000 7fe609f15640 10 monclient: _renew_subs 2024-03-25T18:31:50.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.404+0000 7fe609f15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:31:50.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.404+0000 7fe609f15640 1 -- 172.21.15.3:0/2739526003 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6041e3050 con 0x7fe6040b4e70 2024-03-25T18:31:50.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.404+0000 7fe609f15640 1 librados: init done 2024-03-25T18:31:50.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.405+0000 7fe609f15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:50.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.405+0000 7fe609f15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:50.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.405+0000 7fe609f15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:31:50.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.405+0000 7fe609f15640 1 -- 172.21.15.3:0/2739526003 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe604141540 con 0x7fe6040b4e70 2024-03-25T18:31:50.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.427+0000 7fe5ebfff640 1 -- 172.21.15.3:0/2739526003 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5fc004b60 con 0x7fe6040b4e70 2024-03-25T18:31:50.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.427+0000 7fe5ebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:50.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.427+0000 7fe5ebfff640 1 -- 172.21.15.3:0/2739526003 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fe5fc004cd0 con 0x7fe6040b4e70 2024-03-25T18:31:50.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.427+0000 7fe5ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:50.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.427+0000 7fe5ebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:50.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.427+0000 7fe5ebfff640 10 monclient: dump: 2024-03-25T18:31:50.428 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:50.428 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:50.429 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:50.429 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:50.429 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:50.429 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:50.429 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:50.429 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:50.429 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:50.429 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:50.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.427+0000 7fe601ffb640 4 set_mon_vals no callback set 2024-03-25T18:31:50.430 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.427+0000 7fe5ebfff640 1 -- 172.21.15.3:0/2739526003 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7fe5fc01aca0 con 0x7fe6040b4e70 2024-03-25T18:31:50.430 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.428+0000 7fe6037fe640 10 monclient: get_auth_request con 0x7fe5f006ddd0 auth_method 0 2024-03-25T18:31:50.430 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.428+0000 7fe5ebfff640 10 client.4539.objecter ms_handle_connect 0x7fe5f006ddd0 2024-03-25T18:31:50.434 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.433+0000 7fe5ebfff640 1 -- 172.21.15.3:0/2739526003 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7fe5fc054980 con 0x7fe6040b4e70 2024-03-25T18:31:50.434 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.433+0000 7fe5ebfff640 10 client.4539.objecter ms_dispatch 0x7fe604000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:50.434 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.433+0000 7fe5ebfff640 3 client.4539.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:50.434 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.433+0000 7fe5ebfff640 3 client.4539.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:50.434 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.433+0000 7fe5ebfff640 20 client.4539.objecter dump_active .. 0 homeless 2024-03-25T18:31:50.723 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.722+0000 7fe5ebfff640 1 -- 172.21.15.3:0/2739526003 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe5fc0502c0 con 0x7fe6040b4e70 2024-03-25T18:31:50.723 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.722+0000 7fe5ebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:50.723 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.722+0000 7fe5ebfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:50.975 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.974+0000 7fe609f15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:50.975 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.974+0000 7fe609f15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:50.975 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.974+0000 7fe609f15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:31:50.975 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.974+0000 7fe609f15640 1 -- 172.21.15.3:0/2739526003 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe60404f730 con 0x7fe6040b4e70 2024-03-25T18:31:50.997 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.996+0000 7fe5ebfff640 1 -- 172.21.15.3:0/2739526003 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7fe5fc0523c0 con 0x7fe6040b4e70 2024-03-25T18:31:50.997 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.996+0000 7fe5ebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:50.997 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.996+0000 7fe5ebfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:50.998 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:31:50.998 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:51.000 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.998+0000 7fe609f15640 10 librados: watch_flush enter 2024-03-25T18:31:51.000 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.998+0000 7fe609f15640 10 librados: watch_flush exit 2024-03-25T18:31:51.000 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.998+0000 7fe609f15640 20 client.4539.objecter shutdown clearing up homeless session... 2024-03-25T18:31:51.000 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.998+0000 7fe609f15640 10 client.4539.objecter successfully canceled tick 2024-03-25T18:31:51.000 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.998+0000 7fe609f15640 1 -- 172.21.15.3:0/2739526003 >> v1:172.21.15.3:6800/4195140966 conn(0x7fe5f006ddd0 legacy=0x7fe5f0070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:51.000 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.998+0000 7fe609f15640 10 monclient: shutdown 2024-03-25T18:31:51.000 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.998+0000 7fe609f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:51.000 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.998+0000 7fe609f15640 1 -- 172.21.15.3:0/2739526003 >> v1:172.21.15.3:6789/0 conn(0x7fe6040b4e70 legacy=0x7fe604141f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:51.000 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.998+0000 7fe609f15640 1 -- 172.21.15.3:0/2739526003 shutdown_connections 2024-03-25T18:31:51.000 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.998+0000 7fe609f15640 1 -- 172.21.15.3:0/2739526003 >> 172.21.15.3:0/2739526003 conn(0x7fe6040617a0 msgr2=0x7fe60413e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:51.001 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.998+0000 7fe609f15640 1 -- 172.21.15.3:0/2739526003 shutdown_connections 2024-03-25T18:31:51.001 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.999+0000 7fe609f15640 1 -- 172.21.15.3:0/2739526003 wait complete. 2024-03-25T18:31:51.001 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:50.999+0000 7fe609f15640 1 librados: shutdown 2024-03-25T18:31:51.011 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:57.817 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:57.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.916+0000 7fe1f8a55640 10 monclient: get_monmap_and_config 2024-03-25T18:31:57.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.916+0000 7fe1f8a55640 10 monclient: build_initial_monmap 2024-03-25T18:31:57.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.916+0000 7fe1f8a55640 10 monclient: monmap: 2024-03-25T18:31:57.918 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:31:57.918 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:57.918 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:31:57.917688+0000 2024-03-25T18:31:57.918 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:31:57.917688+0000 2024-03-25T18:31:57.918 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:57.918 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:57.918 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:31:57.918 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:31:57.918 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:31:57.918 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:57.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.917+0000 7fe1f8a55640 1 Processor -- start 2024-03-25T18:31:57.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.917+0000 7fe1f8a55640 1 -- start start 2024-03-25T18:31:57.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.917+0000 7fe1f8a55640 10 monclient: init 2024-03-25T18:31:57.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.917+0000 7fe1f8a55640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:57.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.917+0000 7fe1f8a55640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:31:57.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.917+0000 7fe1f8a55640 10 monclient(hunting): picked mon.noname-c con 0x7fe1f4061830 addr v1:172.21.15.16:6789/0 2024-03-25T18:31:57.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.917+0000 7fe1f8a55640 10 monclient(hunting): picked mon.noname-a con 0x7fe1f413fc20 addr v1:172.21.15.3:6789/0 2024-03-25T18:31:57.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.917+0000 7fe1f8a55640 10 monclient(hunting): picked mon.noname-b con 0x7fe1f4142810 addr v1:172.21.15.3:6790/0 2024-03-25T18:31:57.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.917+0000 7fe1f8a55640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f406ea90 con 0x7fe1f413fc20 2024-03-25T18:31:57.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.917+0000 7fe1f8a55640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f406e7b0 con 0x7fe1f4061830 2024-03-25T18:31:57.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.917+0000 7fe1f8a55640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f4142e60 con 0x7fe1f4142810 2024-03-25T18:31:57.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.917+0000 7fe1f8a55640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:57.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.917+0000 7fe1f8a55640 10 monclient(hunting): authenticate will time out at 7772.178278s 2024-03-25T18:31:57.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.918+0000 7fe1f2d9d640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fe1f4142810 0x7fe1f4145fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:46672/0 (socket says 172.21.15.3:46672) 2024-03-25T18:31:57.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.918+0000 7fe1f2d9d640 1 -- 172.21.15.3:0/858340901 learned_addr learned my addr 172.21.15.3:0/858340901 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:31:57.968 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.967+0000 7fe1f1d9b640 1 -- 172.21.15.3:0/858340901 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2476172100 0 0) 0x7fe1f406ea90 con 0x7fe1f413fc20 2024-03-25T18:31:57.969 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.967+0000 7fe1f1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:31:57.969 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.967+0000 7fe1f1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:57.969 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.967+0000 7fe1f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:57.969 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.967+0000 7fe1f1d9b640 10 monclient(hunting): my global_id is 4542 2024-03-25T18:31:57.969 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.967+0000 7fe1f1d9b640 1 -- 172.21.15.3:0/858340901 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1e0003520 con 0x7fe1f413fc20 2024-03-25T18:31:57.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.975+0000 7fe1f1d9b640 1 -- 172.21.15.3:0/858340901 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3186078881 0 0) 0x7fe1e0003520 con 0x7fe1f413fc20 2024-03-25T18:31:57.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.975+0000 7fe1f1d9b640 1 -- 172.21.15.3:0/858340901 >> v1:172.21.15.3:6790/0 conn(0x7fe1f4142810 legacy=0x7fe1f4145fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:57.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.975+0000 7fe1f1d9b640 1 -- 172.21.15.3:0/858340901 >> v1:172.21.15.16:6789/0 conn(0x7fe1f4061830 legacy=0x7fe1f413f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:57.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.975+0000 7fe1f1d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:31:57.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.975+0000 7fe1f1d9b640 1 monclient: found mon.noname-a 2024-03-25T18:31:57.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.975+0000 7fe1f1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:57.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.975+0000 7fe1f1d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:31:57.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.975+0000 7fe1f1d9b640 1 -- 172.21.15.3:0/858340901 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f4151c70 con 0x7fe1f413fc20 2024-03-25T18:31:57.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.975+0000 7fe1f1d9b640 10 monclient: _finish_auth 0 2024-03-25T18:31:57.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.975+0000 7fe1f1d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:31:57.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.975+0000 7fe1f1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:57.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.975+0000 7fe1f1d9b640 1 -- 172.21.15.3:0/858340901 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fe1dc001690 con 0x7fe1f413fc20 2024-03-25T18:31:57.978 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.975+0000 7fe1f1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:57.978 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.975+0000 7fe1f1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:57.978 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.975+0000 7fe1f1d9b640 10 monclient: dump: 2024-03-25T18:31:57.978 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:57.978 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:57.978 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:57.978 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:57.978 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:57.978 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:57.978 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:57.978 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:57.978 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:57.978 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:57.978 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.975+0000 7fe1f8a55640 5 monclient: authenticate success, global_id 4542 2024-03-25T18:31:57.979 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.975+0000 7fe1f8a55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:57.998 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.997+0000 7fe1f1d9b640 1 -- 172.21.15.3:0/858340901 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1dc0017f0 con 0x7fe1f413fc20 2024-03-25T18:31:57.998 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.997+0000 7fe1f1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:57.999 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.997+0000 7fe1f1d9b640 1 -- 172.21.15.3:0/858340901 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fe1dc001b80 con 0x7fe1f413fc20 2024-03-25T18:31:57.999 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.997+0000 7fe1f1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:57.999 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.997+0000 7fe1f1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:57.999 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.997+0000 7fe1f1d9b640 10 monclient: dump: 2024-03-25T18:31:57.999 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:57.999 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:57.999 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:57.999 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:57.999 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:57.999 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:57.999 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:57.999 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:57.999 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:57.999 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:57.999 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.997+0000 7fe1f8a55640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:58.000 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.997+0000 7fe1f8a55640 4 set_mon_vals no callback set 2024-03-25T18:31:58.001 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.998+0000 7fe1f8a55640 10 monclient: shutdown 2024-03-25T18:31:58.001 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.998+0000 7fe1f8a55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:58.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.998+0000 7fe1f8a55640 1 -- 172.21.15.3:0/858340901 >> v1:172.21.15.3:6789/0 conn(0x7fe1f413fc20 legacy=0x7fe1f4145890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:58.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.998+0000 7fe1f8a55640 1 -- 172.21.15.3:0/858340901 shutdown_connections 2024-03-25T18:31:58.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.998+0000 7fe1f8a55640 1 -- 172.21.15.3:0/858340901 >> 172.21.15.3:0/858340901 conn(0x7fe1f406b1d0 msgr2=0x7fe1f40b4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:58.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.998+0000 7fe1f8a55640 1 -- 172.21.15.3:0/858340901 shutdown_connections 2024-03-25T18:31:58.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.998+0000 7fe1f8a55640 1 -- 172.21.15.3:0/858340901 wait complete. 2024-03-25T18:31:58.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.999+0000 7fe1f8a55640 10 monclient: build_initial_monmap 2024-03-25T18:31:58.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.999+0000 7fe1f8a55640 10 monclient: monmap: 2024-03-25T18:31:58.002 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:31:58.002 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:58.002 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:31:58.002 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:31:58.002 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:58.002 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:31:58.002 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:31:58.003 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:31:58.003 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:31:58.003 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:58.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.999+0000 7fe1f8a55640 1 librados: starting msgr at 2024-03-25T18:31:58.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.999+0000 7fe1f8a55640 1 librados: starting objecter 2024-03-25T18:31:58.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.999+0000 7fe1f8a55640 1 Processor -- start 2024-03-25T18:31:58.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.999+0000 7fe1f8a55640 1 -- start start 2024-03-25T18:31:58.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.999+0000 7fe1f8a55640 1 librados: setting wanted keys 2024-03-25T18:31:58.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.999+0000 7fe1f8a55640 1 librados: calling monclient init 2024-03-25T18:31:58.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:57.999+0000 7fe1f8a55640 10 monclient: init 2024-03-25T18:31:58.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.000+0000 7fe1f8a55640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:58.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.000+0000 7fe1f8a55640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:31:58.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.000+0000 7fe1f8a55640 10 monclient(hunting): picked mon.noname-a con 0x7fe1f4061830 addr v1:172.21.15.3:6789/0 2024-03-25T18:31:58.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.000+0000 7fe1f8a55640 10 monclient(hunting): picked mon.noname-b con 0x7fe1f413fc20 addr v1:172.21.15.16:6789/0 2024-03-25T18:31:58.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.000+0000 7fe1f8a55640 10 monclient(hunting): picked mon.noname-c con 0x7fe1f4142810 addr v1:172.21.15.3:6790/0 2024-03-25T18:31:58.004 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.000+0000 7fe1f8a55640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f413a090 con 0x7fe1f4061830 2024-03-25T18:31:58.004 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.000+0000 7fe1f8a55640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f413acf0 con 0x7fe1f413fc20 2024-03-25T18:31:58.004 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.000+0000 7fe1f8a55640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f413afd0 con 0x7fe1f4142810 2024-03-25T18:31:58.004 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.000+0000 7fe1f8a55640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:58.004 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.000+0000 7fe1f8a55640 10 monclient(hunting): authenticate will time out at 7772.260855s 2024-03-25T18:31:58.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.006+0000 7fe1ebfff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe1f413fc20 0x7fe1f41449c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:47280/0 (socket says 172.21.15.3:47280) 2024-03-25T18:31:58.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.006+0000 7fe1ebfff640 1 -- 172.21.15.3:0/3167120938 learned_addr learned my addr 172.21.15.3:0/3167120938 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:31:58.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.007+0000 7fe1eaffd640 10 client.?.objecter ms_handle_connect 0x7fe1f413fc20 2024-03-25T18:31:58.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.007+0000 7fe1eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:58.015 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.014+0000 7fe1eaffd640 10 client.?.objecter ms_handle_connect 0x7fe1f4061830 2024-03-25T18:31:58.015 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.014+0000 7fe1eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:58.021 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.020+0000 7fe1eaffd640 1 -- 172.21.15.3:0/3167120938 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1353497157 0 0) 0x7fe1f413a090 con 0x7fe1f4061830 2024-03-25T18:31:58.021 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.020+0000 7fe1eaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:58.021 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.020+0000 7fe1eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:58.021 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.020+0000 7fe1eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:58.021 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.020+0000 7fe1eaffd640 10 monclient(hunting): my global_id is 4545 2024-03-25T18:31:58.021 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.020+0000 7fe1eaffd640 1 -- 172.21.15.3:0/3167120938 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1d4004eb0 con 0x7fe1f4061830 2024-03-25T18:31:58.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.032+0000 7fe1eaffd640 1 -- 172.21.15.3:0/3167120938 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1821921880 0 0) 0x7fe1d4004eb0 con 0x7fe1f4061830 2024-03-25T18:31:58.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.032+0000 7fe1eaffd640 1 -- 172.21.15.3:0/3167120938 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1f413a090 con 0x7fe1f4061830 2024-03-25T18:31:58.034 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.032+0000 7fe1eaffd640 1 -- 172.21.15.3:0/3167120938 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fe1e40018c0 con 0x7fe1f4061830 2024-03-25T18:31:58.034 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.032+0000 7fe1eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:58.034 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.032+0000 7fe1eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:58.034 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.032+0000 7fe1eaffd640 10 monclient(hunting): dump: 2024-03-25T18:31:58.034 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:58.034 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:58.034 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:58.034 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:58.034 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:58.034 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:58.034 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:58.034 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:58.034 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:58.034 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:58.034 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.032+0000 7fe1eaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:58.035 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.034+0000 7fe1eaffd640 10 client.?.objecter ms_handle_connect 0x7fe1f4142810 2024-03-25T18:31:58.035 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.034+0000 7fe1eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:58.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.110+0000 7fe1eaffd640 1 -- 172.21.15.3:0/3167120938 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2355179871 0 0) 0x7fe1f413afd0 con 0x7fe1f4142810 2024-03-25T18:31:58.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.110+0000 7fe1eaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:58.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.110+0000 7fe1eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:58.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.110+0000 7fe1eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:58.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.110+0000 7fe1eaffd640 10 monclient(hunting): my global_id is 4502 2024-03-25T18:31:58.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.110+0000 7fe1eaffd640 1 -- 172.21.15.3:0/3167120938 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1d4004eb0 con 0x7fe1f4142810 2024-03-25T18:31:58.119 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.117+0000 7fe1eaffd640 1 -- 172.21.15.3:0/3167120938 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1426134823 0 0) 0x7fe1f413a090 con 0x7fe1f4061830 2024-03-25T18:31:58.119 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.117+0000 7fe1eaffd640 1 -- 172.21.15.3:0/3167120938 >> v1:172.21.15.3:6790/0 conn(0x7fe1f4142810 legacy=0x7fe1f41450d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:58.119 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.117+0000 7fe1eaffd640 1 -- 172.21.15.3:0/3167120938 >> v1:172.21.15.16:6789/0 conn(0x7fe1f413fc20 legacy=0x7fe1f41449c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:58.119 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.117+0000 7fe1eaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:31:58.119 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.117+0000 7fe1eaffd640 1 monclient: found mon.a 2024-03-25T18:31:58.119 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.117+0000 7fe1eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:58.119 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.117+0000 7fe1eaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:31:58.119 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.117+0000 7fe1eaffd640 1 -- 172.21.15.3:0/3167120938 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f413b2b0 con 0x7fe1f4061830 2024-03-25T18:31:58.119 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.117+0000 7fe1eaffd640 10 monclient: _finish_auth 0 2024-03-25T18:31:58.119 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.117+0000 7fe1eaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:31:58.119 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.117+0000 7fe1eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:58.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.117+0000 7fe1f8a55640 5 monclient: authenticate success, global_id 4545 2024-03-25T18:31:58.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.117+0000 7fe1f8a55640 10 monclient: _renew_subs 2024-03-25T18:31:58.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.117+0000 7fe1f8a55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:31:58.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.117+0000 7fe1f8a55640 1 -- 172.21.15.3:0/3167120938 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1f4139c40 con 0x7fe1f4061830 2024-03-25T18:31:58.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.117+0000 7fe1f8a55640 10 client.4545.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:58.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.117+0000 7fe1f8a55640 10 monclient: _renew_subs 2024-03-25T18:31:58.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.117+0000 7fe1f8a55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:31:58.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.117+0000 7fe1f8a55640 1 -- 172.21.15.3:0/3167120938 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1f41e3210 con 0x7fe1f4061830 2024-03-25T18:31:58.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.117+0000 7fe1f8a55640 1 librados: init done 2024-03-25T18:31:58.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.118+0000 7fe1f8a55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:58.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.118+0000 7fe1f8a55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:58.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.118+0000 7fe1f8a55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:31:58.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.118+0000 7fe1f8a55640 1 -- 172.21.15.3:0/3167120938 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1f404f730 con 0x7fe1f4061830 2024-03-25T18:31:58.430 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.428+0000 7fe1eaffd640 1 -- 172.21.15.3:0/3167120938 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1e40035f0 con 0x7fe1f4061830 2024-03-25T18:31:58.430 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.428+0000 7fe1eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:58.430 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.428+0000 7fe1eaffd640 1 -- 172.21.15.3:0/3167120938 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fe1e40039c0 con 0x7fe1f4061830 2024-03-25T18:31:58.430 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.428+0000 7fe1eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:58.430 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.428+0000 7fe1eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:58.430 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.428+0000 7fe1eaffd640 10 monclient: dump: 2024-03-25T18:31:58.430 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:31:58.430 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:31:58.430 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:58.430 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:31:58.431 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:58.431 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:31:58.431 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:31:58.431 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:31:58.431 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:31:58.431 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:31:58.431 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.428+0000 7fe1f159a640 4 set_mon_vals no callback set 2024-03-25T18:31:58.431 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.429+0000 7fe1eaffd640 1 -- 172.21.15.3:0/3167120938 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7fe1e4003be0 con 0x7fe1f4061830 2024-03-25T18:31:58.432 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.430+0000 7fe1ebfff640 10 monclient: get_auth_request con 0x7fe1d406ddd0 auth_method 0 2024-03-25T18:31:58.432 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.430+0000 7fe1eaffd640 10 client.4545.objecter ms_handle_connect 0x7fe1d406ddd0 2024-03-25T18:31:58.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.436+0000 7fe1eaffd640 1 -- 172.21.15.3:0/3167120938 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7fe1e4053640 con 0x7fe1f4061830 2024-03-25T18:31:58.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.436+0000 7fe1eaffd640 10 client.4545.objecter ms_dispatch 0x7fe1f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:58.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.436+0000 7fe1eaffd640 3 client.4545.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:58.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.436+0000 7fe1eaffd640 3 client.4545.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:58.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.436+0000 7fe1eaffd640 20 client.4545.objecter dump_active .. 0 homeless 2024-03-25T18:31:58.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.436+0000 7fe1eaffd640 1 -- 172.21.15.3:0/3167120938 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe1e404f150 con 0x7fe1f4061830 2024-03-25T18:31:58.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.436+0000 7fe1eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:58.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.436+0000 7fe1eaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:58.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.696+0000 7fe1f8a55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:58.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.696+0000 7fe1f8a55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:58.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.696+0000 7fe1f8a55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:31:58.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.696+0000 7fe1f8a55640 1 -- 172.21.15.3:0/3167120938 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1f40b0680 con 0x7fe1f4061830 2024-03-25T18:31:58.735 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.734+0000 7fe1eaffd640 1 -- 172.21.15.3:0/3167120938 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7fe1e4051080 con 0x7fe1f4061830 2024-03-25T18:31:58.735 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.734+0000 7fe1eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:58.735 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.734+0000 7fe1eaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:58.735 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:31:58.735 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:58.738 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.736+0000 7fe1f8a55640 10 librados: watch_flush enter 2024-03-25T18:31:58.738 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.736+0000 7fe1f8a55640 10 librados: watch_flush exit 2024-03-25T18:31:58.738 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.736+0000 7fe1f8a55640 20 client.4545.objecter shutdown clearing up homeless session... 2024-03-25T18:31:58.738 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.736+0000 7fe1f8a55640 10 client.4545.objecter successfully canceled tick 2024-03-25T18:31:58.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.736+0000 7fe1f8a55640 1 -- 172.21.15.3:0/3167120938 >> v1:172.21.15.3:6800/4195140966 conn(0x7fe1d406ddd0 legacy=0x7fe1d4070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:58.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.736+0000 7fe1f8a55640 10 monclient: shutdown 2024-03-25T18:31:58.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.736+0000 7fe1f8a55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:58.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.736+0000 7fe1f8a55640 1 -- 172.21.15.3:0/3167120938 >> v1:172.21.15.3:6789/0 conn(0x7fe1f4061830 legacy=0x7fe1f4139530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:58.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.736+0000 7fe1f8a55640 1 -- 172.21.15.3:0/3167120938 shutdown_connections 2024-03-25T18:31:58.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.736+0000 7fe1f8a55640 1 -- 172.21.15.3:0/3167120938 >> 172.21.15.3:0/3167120938 conn(0x7fe1f406b1d0 msgr2=0x7fe1f40adfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:58.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.736+0000 7fe1f8a55640 1 -- 172.21.15.3:0/3167120938 shutdown_connections 2024-03-25T18:31:58.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.736+0000 7fe1f8a55640 1 -- 172.21.15.3:0/3167120938 wait complete. 2024-03-25T18:31:58.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:31:58.736+0000 7fe1f8a55640 1 librados: shutdown 2024-03-25T18:31:58.748 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:02.637 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:07:16.366 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:02.652 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:07:16.381 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:02.678 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:07:16.407 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:02.692 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:07:16.421 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:02.705 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:07:16.397 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:02.706 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:07:16.435 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:02.711 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:07:16.403 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:02.732 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:07:16.461 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:02.743 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:07:16.435 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:05.556 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:05.674 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.673+0000 7fbc88d77640 10 monclient: get_monmap_and_config 2024-03-25T18:32:05.675 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.673+0000 7fbc88d77640 10 monclient: build_initial_monmap 2024-03-25T18:32:05.675 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.673+0000 7fbc88d77640 10 monclient: monmap: 2024-03-25T18:32:05.675 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:32:05.675 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:05.675 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:32:05.674763+0000 2024-03-25T18:32:05.675 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:32:05.674763+0000 2024-03-25T18:32:05.675 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:05.675 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:05.675 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:32:05.675 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:32:05.675 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:32:05.675 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:05.676 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.674+0000 7fbc88d77640 1 Processor -- start 2024-03-25T18:32:05.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.674+0000 7fbc88d77640 1 -- start start 2024-03-25T18:32:05.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.674+0000 7fbc88d77640 10 monclient: init 2024-03-25T18:32:05.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.675+0000 7fbc88d77640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:05.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.675+0000 7fbc88d77640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:32:05.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.675+0000 7fbc88d77640 10 monclient(hunting): picked mon.noname-c con 0x7fbc840b1a70 addr v1:172.21.15.16:6789/0 2024-03-25T18:32:05.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.675+0000 7fbc88d77640 10 monclient(hunting): picked mon.noname-b con 0x7fbc841405b0 addr v1:172.21.15.3:6790/0 2024-03-25T18:32:05.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.675+0000 7fbc88d77640 10 monclient(hunting): picked mon.noname-a con 0x7fbc8413d690 addr v1:172.21.15.3:6789/0 2024-03-25T18:32:05.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.675+0000 7fbc88d77640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc84072990 con 0x7fbc8413d690 2024-03-25T18:32:05.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.675+0000 7fbc88d77640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc84072c70 con 0x7fbc840b1a70 2024-03-25T18:32:05.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.675+0000 7fbc88d77640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc8413dc30 con 0x7fbc841405b0 2024-03-25T18:32:05.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.675+0000 7fbc88d77640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:05.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.675+0000 7fbc88d77640 10 monclient(hunting): authenticate will time out at 7779.936033s 2024-03-25T18:32:05.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.688+0000 7fbc81d9b640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fbc841405b0 0x7fbc8406ffb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:38162/0 (socket says 172.21.15.3:38162) 2024-03-25T18:32:05.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.688+0000 7fbc81d9b640 1 -- 172.21.15.3:0/1875232806 learned_addr learned my addr 172.21.15.3:0/1875232806 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:32:05.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.757+0000 7fbc8159a640 1 -- 172.21.15.3:0/1875232806 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3492108030 0 0) 0x7fbc84072990 con 0x7fbc8413d690 2024-03-25T18:32:05.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.757+0000 7fbc8159a640 10 monclient(hunting): _negotiate 2024-03-25T18:32:05.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.757+0000 7fbc8159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:05.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.757+0000 7fbc8159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:05.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.757+0000 7fbc8159a640 10 monclient(hunting): my global_id is 4548 2024-03-25T18:32:05.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.757+0000 7fbc8159a640 1 -- 172.21.15.3:0/1875232806 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc6c003520 con 0x7fbc8413d690 2024-03-25T18:32:05.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.762+0000 7fbc8159a640 1 -- 172.21.15.3:0/1875232806 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2308624360 0 0) 0x7fbc6c003520 con 0x7fbc8413d690 2024-03-25T18:32:05.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.762+0000 7fbc8159a640 1 -- 172.21.15.3:0/1875232806 >> v1:172.21.15.3:6790/0 conn(0x7fbc841405b0 legacy=0x7fbc8406ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:05.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.762+0000 7fbc8159a640 1 -- 172.21.15.3:0/1875232806 >> v1:172.21.15.16:6789/0 conn(0x7fbc840b1a70 legacy=0x7fbc8413cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:05.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.762+0000 7fbc8159a640 10 monclient: _finish_hunting 0 2024-03-25T18:32:05.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.762+0000 7fbc8159a640 1 monclient: found mon.noname-a 2024-03-25T18:32:05.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.762+0000 7fbc8159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:05.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.762+0000 7fbc8159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:32:05.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.762+0000 7fbc8159a640 1 -- 172.21.15.3:0/1875232806 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc84140b20 con 0x7fbc8413d690 2024-03-25T18:32:05.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.762+0000 7fbc8159a640 10 monclient: _finish_auth 0 2024-03-25T18:32:05.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.762+0000 7fbc8159a640 10 monclient: _check_auth_tickets 2024-03-25T18:32:05.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.762+0000 7fbc8159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:05.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.762+0000 7fbc8159a640 1 -- 172.21.15.3:0/1875232806 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fbc7c001a10 con 0x7fbc8413d690 2024-03-25T18:32:05.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.762+0000 7fbc8159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:05.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.762+0000 7fbc8159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:05.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.762+0000 7fbc8159a640 10 monclient: dump: 2024-03-25T18:32:05.765 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:05.765 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:05.765 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:05.765 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:05.765 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:05.765 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:05.765 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:05.765 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:05.765 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:05.765 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:05.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.762+0000 7fbc88d77640 5 monclient: authenticate success, global_id 4548 2024-03-25T18:32:05.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.762+0000 7fbc88d77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:05.797 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.796+0000 7fbc8159a640 1 -- 172.21.15.3:0/1875232806 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc7c001b70 con 0x7fbc8413d690 2024-03-25T18:32:05.797 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.796+0000 7fbc8159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:05.798 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.796+0000 7fbc8159a640 1 -- 172.21.15.3:0/1875232806 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fbc7c001f80 con 0x7fbc8413d690 2024-03-25T18:32:05.798 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.796+0000 7fbc8159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:05.798 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.796+0000 7fbc8159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:05.798 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.796+0000 7fbc8159a640 10 monclient: dump: 2024-03-25T18:32:05.798 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:05.798 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:05.798 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:05.798 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:05.798 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:05.798 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:05.798 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:05.798 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:05.799 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:05.799 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:05.799 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.796+0000 7fbc88d77640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:05.799 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.796+0000 7fbc88d77640 4 set_mon_vals no callback set 2024-03-25T18:32:05.799 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.797+0000 7fbc88d77640 10 monclient: shutdown 2024-03-25T18:32:05.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.797+0000 7fbc88d77640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:05.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.797+0000 7fbc88d77640 1 -- 172.21.15.3:0/1875232806 >> v1:172.21.15.3:6789/0 conn(0x7fbc8413d690 legacy=0x7fbc8414ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:05.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.797+0000 7fbc88d77640 1 -- 172.21.15.3:0/1875232806 shutdown_connections 2024-03-25T18:32:05.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.797+0000 7fbc88d77640 1 -- 172.21.15.3:0/1875232806 >> 172.21.15.3:0/1875232806 conn(0x7fbc840b28f0 msgr2=0x7fbc840b2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:05.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.798+0000 7fbc88d77640 1 -- 172.21.15.3:0/1875232806 shutdown_connections 2024-03-25T18:32:05.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.798+0000 7fbc88d77640 1 -- 172.21.15.3:0/1875232806 wait complete. 2024-03-25T18:32:05.801 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.799+0000 7fbc88d77640 10 monclient: build_initial_monmap 2024-03-25T18:32:05.801 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.799+0000 7fbc88d77640 10 monclient: monmap: 2024-03-25T18:32:05.801 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:32:05.801 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:05.801 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:32:05.801 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:32:05.801 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:05.801 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:32:05.801 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:32:05.801 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:32:05.801 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:32:05.801 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:05.802 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.799+0000 7fbc88d77640 1 librados: starting msgr at 2024-03-25T18:32:05.802 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.799+0000 7fbc88d77640 1 librados: starting objecter 2024-03-25T18:32:05.802 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.799+0000 7fbc88d77640 1 Processor -- start 2024-03-25T18:32:05.802 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.800+0000 7fbc88d77640 1 -- start start 2024-03-25T18:32:05.802 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.800+0000 7fbc88d77640 1 librados: setting wanted keys 2024-03-25T18:32:05.802 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.800+0000 7fbc88d77640 1 librados: calling monclient init 2024-03-25T18:32:05.802 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.800+0000 7fbc88d77640 10 monclient: init 2024-03-25T18:32:05.803 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.800+0000 7fbc88d77640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:05.803 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.800+0000 7fbc88d77640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:32:05.803 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.800+0000 7fbc88d77640 10 monclient(hunting): picked mon.noname-a con 0x7fbc840b1a70 addr v1:172.21.15.3:6789/0 2024-03-25T18:32:05.803 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.800+0000 7fbc88d77640 10 monclient(hunting): picked mon.noname-b con 0x7fbc8413d690 addr v1:172.21.15.16:6789/0 2024-03-25T18:32:05.803 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.800+0000 7fbc88d77640 10 monclient(hunting): picked mon.noname-c con 0x7fbc841405b0 addr v1:172.21.15.3:6790/0 2024-03-25T18:32:05.803 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.801+0000 7fbc88d77640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc841415f0 con 0x7fbc840b1a70 2024-03-25T18:32:05.803 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.801+0000 7fbc88d77640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc840af5c0 con 0x7fbc8413d690 2024-03-25T18:32:05.803 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.801+0000 7fbc88d77640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc8406bf20 con 0x7fbc841405b0 2024-03-25T18:32:05.804 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.801+0000 7fbc88d77640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:05.804 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.801+0000 7fbc88d77640 10 monclient(hunting): authenticate will time out at 7780.061417s 2024-03-25T18:32:05.823 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.822+0000 7fbc8259c640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fbc840b1a70 0x7fbc841d78a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:48570/0 (socket says 172.21.15.3:48570) 2024-03-25T18:32:05.824 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.822+0000 7fbc8259c640 1 -- 172.21.15.3:0/122654817 learned_addr learned my addr 172.21.15.3:0/122654817 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:32:05.824 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.823+0000 7fbc72ffd640 10 client.?.objecter ms_handle_connect 0x7fbc840b1a70 2024-03-25T18:32:05.824 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.823+0000 7fbc72ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:05.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.829+0000 7fbc72ffd640 10 client.?.objecter ms_handle_connect 0x7fbc841405b0 2024-03-25T18:32:05.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.829+0000 7fbc72ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:05.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.840+0000 7fbc72ffd640 10 client.?.objecter ms_handle_connect 0x7fbc8413d690 2024-03-25T18:32:05.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.840+0000 7fbc72ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:05.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.844+0000 7fbc72ffd640 1 -- 172.21.15.3:0/122654817 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3386644318 0 0) 0x7fbc8406bf20 con 0x7fbc841405b0 2024-03-25T18:32:05.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.844+0000 7fbc72ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:05.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.844+0000 7fbc72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:05.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.844+0000 7fbc72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:05.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.844+0000 7fbc72ffd640 10 monclient(hunting): my global_id is 4508 2024-03-25T18:32:05.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.844+0000 7fbc72ffd640 1 -- 172.21.15.3:0/122654817 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc64004eb0 con 0x7fbc841405b0 2024-03-25T18:32:05.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.845+0000 7fbc72ffd640 1 -- 172.21.15.3:0/122654817 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3921468391 0 0) 0x7fbc841415f0 con 0x7fbc840b1a70 2024-03-25T18:32:05.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.845+0000 7fbc72ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:05.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.845+0000 7fbc72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:05.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.845+0000 7fbc72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:05.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.845+0000 7fbc72ffd640 10 monclient(hunting): my global_id is 4551 2024-03-25T18:32:05.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.845+0000 7fbc72ffd640 1 -- 172.21.15.3:0/122654817 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc8406bf20 con 0x7fbc840b1a70 2024-03-25T18:32:05.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.846+0000 7fbc72ffd640 1 -- 172.21.15.3:0/122654817 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2198898128 0 0) 0x7fbc840af5c0 con 0x7fbc8413d690 2024-03-25T18:32:05.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.846+0000 7fbc72ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:05.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.846+0000 7fbc72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:05.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.846+0000 7fbc72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:05.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.846+0000 7fbc72ffd640 10 monclient(hunting): my global_id is 4504 2024-03-25T18:32:05.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.846+0000 7fbc72ffd640 1 -- 172.21.15.3:0/122654817 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc841415f0 con 0x7fbc8413d690 2024-03-25T18:32:05.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.871+0000 7fbc72ffd640 1 -- 172.21.15.3:0/122654817 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3131934119 0 0) 0x7fbc64004eb0 con 0x7fbc841405b0 2024-03-25T18:32:05.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.871+0000 7fbc72ffd640 1 -- 172.21.15.3:0/122654817 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc840af5c0 con 0x7fbc841405b0 2024-03-25T18:32:05.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.871+0000 7fbc72ffd640 1 -- 172.21.15.3:0/122654817 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fbc7c001f40 con 0x7fbc841405b0 2024-03-25T18:32:05.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.871+0000 7fbc72ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:05.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.871+0000 7fbc72ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:32:05.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.871+0000 7fbc72ffd640 10 monclient(hunting): dump: 2024-03-25T18:32:05.873 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:05.873 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:05.873 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:05.873 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:05.873 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:05.873 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:05.873 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:05.873 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:05.873 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:05.873 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:05.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.871+0000 7fbc72ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:05.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.883+0000 7fbc72ffd640 1 -- 172.21.15.3:0/122654817 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3844953435 0 0) 0x7fbc841415f0 con 0x7fbc8413d690 2024-03-25T18:32:05.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.883+0000 7fbc72ffd640 1 -- 172.21.15.3:0/122654817 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc64004eb0 con 0x7fbc8413d690 2024-03-25T18:32:05.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.883+0000 7fbc72ffd640 1 -- 172.21.15.3:0/122654817 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fbc780017c0 con 0x7fbc8413d690 2024-03-25T18:32:05.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.883+0000 7fbc72ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:05.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.883+0000 7fbc72ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:05.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.883+0000 7fbc72ffd640 10 monclient(hunting): dump: 2024-03-25T18:32:05.885 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:05.885 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:05.885 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:05.885 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:05.885 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:05.885 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:05.885 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:05.886 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:05.886 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:05.886 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:05.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.890+0000 7fbc72ffd640 1 -- 172.21.15.3:0/122654817 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1203472830 0 0) 0x7fbc840af5c0 con 0x7fbc841405b0 2024-03-25T18:32:05.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.890+0000 7fbc72ffd640 1 -- 172.21.15.3:0/122654817 >> v1:172.21.15.16:6789/0 conn(0x7fbc8413d690 legacy=0x7fbc8406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:05.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.890+0000 7fbc72ffd640 1 -- 172.21.15.3:0/122654817 >> v1:172.21.15.3:6789/0 conn(0x7fbc840b1a70 legacy=0x7fbc841d78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:05.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.890+0000 7fbc72ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:32:05.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.890+0000 7fbc72ffd640 1 monclient: found mon.c 2024-03-25T18:32:05.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.890+0000 7fbc72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:05.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.890+0000 7fbc72ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:32:05.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.890+0000 7fbc72ffd640 1 -- 172.21.15.3:0/122654817 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc8406c0b0 con 0x7fbc841405b0 2024-03-25T18:32:05.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.890+0000 7fbc72ffd640 10 monclient: _finish_auth 0 2024-03-25T18:32:05.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.890+0000 7fbc72ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:32:05.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.890+0000 7fbc72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:05.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.890+0000 7fbc88d77640 5 monclient: authenticate success, global_id 4508 2024-03-25T18:32:05.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.890+0000 7fbc88d77640 10 monclient: _renew_subs 2024-03-25T18:32:05.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.890+0000 7fbc88d77640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:32:05.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.890+0000 7fbc88d77640 1 -- 172.21.15.3:0/122654817 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc84136f20 con 0x7fbc841405b0 2024-03-25T18:32:05.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.890+0000 7fbc88d77640 10 client.4508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:05.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.890+0000 7fbc88d77640 10 monclient: _renew_subs 2024-03-25T18:32:05.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.890+0000 7fbc88d77640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:32:05.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.890+0000 7fbc88d77640 1 -- 172.21.15.3:0/122654817 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc841e3a00 con 0x7fbc841405b0 2024-03-25T18:32:05.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.890+0000 7fbc88d77640 1 librados: init done 2024-03-25T18:32:05.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.891+0000 7fbc88d77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:05.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.891+0000 7fbc88d77640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:05.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.891+0000 7fbc88d77640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:32:05.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:05.891+0000 7fbc88d77640 1 -- 172.21.15.3:0/122654817 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc8406e400 con 0x7fbc841405b0 2024-03-25T18:32:06.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.212+0000 7fbc72ffd640 1 -- 172.21.15.3:0/122654817 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc7c003c20 con 0x7fbc841405b0 2024-03-25T18:32:06.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.212+0000 7fbc72ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:06.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.212+0000 7fbc72ffd640 1 -- 172.21.15.3:0/122654817 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fbc7c003f30 con 0x7fbc841405b0 2024-03-25T18:32:06.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.212+0000 7fbc72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:06.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.212+0000 7fbc72ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:06.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.212+0000 7fbc72ffd640 10 monclient: dump: 2024-03-25T18:32:06.215 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:06.215 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:06.215 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:06.215 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:06.215 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:06.215 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:06.215 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:06.215 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:06.215 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:06.215 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:06.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.212+0000 7fbc80d99640 4 set_mon_vals no callback set 2024-03-25T18:32:06.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.213+0000 7fbc72ffd640 1 -- 172.21.15.3:0/122654817 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7fbc7c019f00 con 0x7fbc841405b0 2024-03-25T18:32:06.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.214+0000 7fbc8259c640 10 monclient: get_auth_request con 0x7fbc6406df70 auth_method 0 2024-03-25T18:32:06.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.214+0000 7fbc72ffd640 10 client.4508.objecter ms_handle_connect 0x7fbc6406df70 2024-03-25T18:32:06.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.220+0000 7fbc72ffd640 1 -- 172.21.15.3:0/122654817 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7fbc7c053a60 con 0x7fbc841405b0 2024-03-25T18:32:06.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.220+0000 7fbc72ffd640 10 client.4508.objecter ms_dispatch 0x7fbc84000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:06.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.220+0000 7fbc72ffd640 3 client.4508.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:06.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.220+0000 7fbc72ffd640 3 client.4508.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:06.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.220+0000 7fbc72ffd640 20 client.4508.objecter dump_active .. 0 homeless 2024-03-25T18:32:06.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.220+0000 7fbc72ffd640 1 -- 172.21.15.3:0/122654817 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbc7c04f570 con 0x7fbc841405b0 2024-03-25T18:32:06.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.220+0000 7fbc72ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:06.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.220+0000 7fbc72ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:06.478 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.477+0000 7fbc88d77640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:06.479 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.477+0000 7fbc88d77640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:06.479 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.477+0000 7fbc88d77640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:32:06.479 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.477+0000 7fbc88d77640 1 -- 172.21.15.3:0/122654817 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc840b37c0 con 0x7fbc841405b0 2024-03-25T18:32:06.510 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.509+0000 7fbc72ffd640 1 -- 172.21.15.3:0/122654817 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7fbc7c0514a0 con 0x7fbc841405b0 2024-03-25T18:32:06.510 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.509+0000 7fbc72ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:06.511 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.509+0000 7fbc72ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:06.511 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:32:06.511 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:06.513 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.511+0000 7fbc88d77640 10 librados: watch_flush enter 2024-03-25T18:32:06.513 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.511+0000 7fbc88d77640 10 librados: watch_flush exit 2024-03-25T18:32:06.513 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.511+0000 7fbc88d77640 20 client.4508.objecter shutdown clearing up homeless session... 2024-03-25T18:32:06.513 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.511+0000 7fbc88d77640 10 client.4508.objecter successfully canceled tick 2024-03-25T18:32:06.513 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.511+0000 7fbc88d77640 1 -- 172.21.15.3:0/122654817 >> v1:172.21.15.3:6800/4195140966 conn(0x7fbc6406df70 legacy=0x7fbc64070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:06.513 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.511+0000 7fbc88d77640 10 monclient: shutdown 2024-03-25T18:32:06.513 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.511+0000 7fbc88d77640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:06.513 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.511+0000 7fbc88d77640 1 -- 172.21.15.3:0/122654817 >> v1:172.21.15.3:6790/0 conn(0x7fbc841405b0 legacy=0x7fbc841e22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:06.513 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.511+0000 7fbc88d77640 1 -- 172.21.15.3:0/122654817 shutdown_connections 2024-03-25T18:32:06.513 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.511+0000 7fbc88d77640 1 -- 172.21.15.3:0/122654817 >> 172.21.15.3:0/122654817 conn(0x7fbc840b28f0 msgr2=0x7fbc8413b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:06.514 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.511+0000 7fbc88d77640 1 -- 172.21.15.3:0/122654817 shutdown_connections 2024-03-25T18:32:06.514 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.511+0000 7fbc88d77640 1 -- 172.21.15.3:0/122654817 wait complete. 2024-03-25T18:32:06.514 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:06.511+0000 7fbc88d77640 1 librados: shutdown 2024-03-25T18:32:06.525 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:13.331 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:13.448 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.446+0000 7ff22c53d640 10 monclient: get_monmap_and_config 2024-03-25T18:32:13.448 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.446+0000 7ff22c53d640 10 monclient: build_initial_monmap 2024-03-25T18:32:13.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.446+0000 7ff22c53d640 10 monclient: monmap: 2024-03-25T18:32:13.449 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:32:13.449 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:13.449 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:32:13.448042+0000 2024-03-25T18:32:13.449 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:32:13.448042+0000 2024-03-25T18:32:13.449 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:13.449 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:13.449 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:32:13.449 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:32:13.449 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:32:13.449 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:13.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.447+0000 7ff22c53d640 1 Processor -- start 2024-03-25T18:32:13.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.447+0000 7ff22c53d640 1 -- start start 2024-03-25T18:32:13.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.447+0000 7ff22c53d640 10 monclient: init 2024-03-25T18:32:13.450 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.448+0000 7ff22c53d640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:13.450 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.448+0000 7ff22c53d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:32:13.451 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.448+0000 7ff22c53d640 10 monclient(hunting): picked mon.noname-a con 0x7ff2240b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:32:13.451 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.448+0000 7ff22c53d640 10 monclient(hunting): picked mon.noname-c con 0x7ff2240b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:32:13.451 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.448+0000 7ff22c53d640 10 monclient(hunting): picked mon.noname-b con 0x7ff22413c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:32:13.451 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.448+0000 7ff22c53d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2240ab0e0 con 0x7ff2240b4ff0 2024-03-25T18:32:13.451 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.448+0000 7ff22c53d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2240ab3c0 con 0x7ff2240b4670 2024-03-25T18:32:13.451 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.448+0000 7ff22c53d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2240a9f30 con 0x7ff22413c480 2024-03-25T18:32:13.451 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.448+0000 7ff22c53d640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:13.451 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.448+0000 7ff22c53d640 10 monclient(hunting): authenticate will time out at 7787.709109s 2024-03-25T18:32:13.458 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.457+0000 7ff229ad8640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff2240b4670 0x7ff2241425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:35906/0 (socket says 172.21.15.3:35906) 2024-03-25T18:32:13.458 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.457+0000 7ff229ad8640 1 -- 172.21.15.3:0/3243676216 learned_addr learned my addr 172.21.15.3:0/3243676216 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:32:13.469 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.468+0000 7ff2292d7640 1 -- 172.21.15.3:0/3243676216 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123474369 0 0) 0x7ff2240ab3c0 con 0x7ff2240b4670 2024-03-25T18:32:13.469 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.468+0000 7ff2292d7640 10 monclient(hunting): _negotiate 2024-03-25T18:32:13.469 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.468+0000 7ff2292d7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:13.469 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.468+0000 7ff2292d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:13.469 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.468+0000 7ff2292d7640 10 monclient(hunting): my global_id is 4507 2024-03-25T18:32:13.469 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.468+0000 7ff2292d7640 1 -- 172.21.15.3:0/3243676216 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff210003520 con 0x7ff2240b4670 2024-03-25T18:32:13.475 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.474+0000 7ff2292d7640 1 -- 172.21.15.3:0/3243676216 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2591413509 0 0) 0x7ff2240ab0e0 con 0x7ff2240b4ff0 2024-03-25T18:32:13.475 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.474+0000 7ff2292d7640 10 monclient(hunting): _negotiate 2024-03-25T18:32:13.475 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.474+0000 7ff2292d7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:13.475 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.474+0000 7ff2292d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:13.476 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.474+0000 7ff2292d7640 10 monclient(hunting): my global_id is 4554 2024-03-25T18:32:13.476 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.474+0000 7ff2292d7640 1 -- 172.21.15.3:0/3243676216 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2240ab3c0 con 0x7ff2240b4ff0 2024-03-25T18:32:13.477 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.476+0000 7ff2292d7640 1 -- 172.21.15.3:0/3243676216 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3008434076 0 0) 0x7ff210003520 con 0x7ff2240b4670 2024-03-25T18:32:13.477 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.476+0000 7ff2292d7640 1 -- 172.21.15.3:0/3243676216 >> v1:172.21.15.3:6790/0 conn(0x7ff22413c480 legacy=0x7ff22414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:13.477 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.476+0000 7ff2292d7640 1 -- 172.21.15.3:0/3243676216 >> v1:172.21.15.3:6789/0 conn(0x7ff2240b4ff0 legacy=0x7ff22413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:13.477 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.476+0000 7ff2292d7640 10 monclient: _finish_hunting 0 2024-03-25T18:32:13.477 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.476+0000 7ff2292d7640 1 monclient: found mon.noname-c 2024-03-25T18:32:13.477 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.476+0000 7ff2292d7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:13.477 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.476+0000 7ff2292d7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:32:13.478 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.476+0000 7ff2292d7640 1 -- 172.21.15.3:0/3243676216 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2240a9b70 con 0x7ff2240b4670 2024-03-25T18:32:13.478 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.476+0000 7ff2292d7640 10 monclient: _finish_auth 0 2024-03-25T18:32:13.478 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.476+0000 7ff2292d7640 10 monclient: _check_auth_tickets 2024-03-25T18:32:13.478 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.476+0000 7ff2292d7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:13.478 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.476+0000 7ff22c53d640 5 monclient: authenticate success, global_id 4507 2024-03-25T18:32:13.478 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.476+0000 7ff22c53d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:13.483 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.482+0000 7ff2292d7640 1 -- 172.21.15.3:0/3243676216 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7ff214001650 con 0x7ff2240b4670 2024-03-25T18:32:13.483 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.482+0000 7ff2292d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:13.483 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.482+0000 7ff2292d7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:32:13.483 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.482+0000 7ff2292d7640 10 monclient: dump: 2024-03-25T18:32:13.483 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:13.483 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:13.483 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:13.483 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:13.483 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:13.483 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:13.483 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:13.483 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:13.484 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:13.484 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:13.484 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.482+0000 7ff2292d7640 1 -- 172.21.15.3:0/3243676216 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2140018a0 con 0x7ff2240b4670 2024-03-25T18:32:13.484 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.482+0000 7ff2292d7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:13.484 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.482+0000 7ff2292d7640 1 -- 172.21.15.3:0/3243676216 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7ff214001c30 con 0x7ff2240b4670 2024-03-25T18:32:13.484 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.482+0000 7ff22c53d640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:13.484 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.482+0000 7ff22c53d640 4 set_mon_vals no callback set 2024-03-25T18:32:13.484 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.482+0000 7ff2292d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:13.484 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.482+0000 7ff2292d7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:13.484 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.482+0000 7ff2292d7640 10 monclient: dump: 2024-03-25T18:32:13.484 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:13.484 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:13.484 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:13.484 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:13.484 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:13.484 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:13.485 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:13.485 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:13.485 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:13.485 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:13.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.483+0000 7ff22c53d640 10 monclient: shutdown 2024-03-25T18:32:13.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.483+0000 7ff22c53d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:13.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.483+0000 7ff22c53d640 1 -- 172.21.15.3:0/3243676216 >> v1:172.21.15.16:6789/0 conn(0x7ff2240b4670 legacy=0x7ff2241425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:13.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.483+0000 7ff22c53d640 1 -- 172.21.15.3:0/3243676216 shutdown_connections 2024-03-25T18:32:13.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.483+0000 7ff22c53d640 1 -- 172.21.15.3:0/3243676216 >> 172.21.15.3:0/3243676216 conn(0x7ff2240617a0 msgr2=0x7ff224061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:13.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.483+0000 7ff22c53d640 1 -- 172.21.15.3:0/3243676216 shutdown_connections 2024-03-25T18:32:13.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.483+0000 7ff22c53d640 1 -- 172.21.15.3:0/3243676216 wait complete. 2024-03-25T18:32:13.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.483+0000 7ff22c53d640 10 monclient: build_initial_monmap 2024-03-25T18:32:13.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.483+0000 7ff22c53d640 10 monclient: monmap: 2024-03-25T18:32:13.488 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:32:13.488 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:13.488 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:32:13.488 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:32:13.488 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:13.488 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:32:13.488 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:32:13.488 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:32:13.488 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:32:13.488 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:13.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.484+0000 7ff22c53d640 1 librados: starting msgr at 2024-03-25T18:32:13.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.484+0000 7ff22c53d640 1 librados: starting objecter 2024-03-25T18:32:13.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.484+0000 7ff22c53d640 1 Processor -- start 2024-03-25T18:32:13.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.484+0000 7ff22c53d640 1 -- start start 2024-03-25T18:32:13.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.484+0000 7ff22c53d640 1 librados: setting wanted keys 2024-03-25T18:32:13.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.484+0000 7ff22c53d640 1 librados: calling monclient init 2024-03-25T18:32:13.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.484+0000 7ff22c53d640 10 monclient: init 2024-03-25T18:32:13.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.484+0000 7ff22c53d640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:13.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.484+0000 7ff22c53d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:32:13.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.484+0000 7ff22c53d640 10 monclient(hunting): picked mon.noname-b con 0x7ff2240b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:32:13.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.484+0000 7ff22c53d640 10 monclient(hunting): picked mon.noname-c con 0x7ff2240b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:32:13.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.484+0000 7ff22c53d640 10 monclient(hunting): picked mon.noname-a con 0x7ff22413c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:32:13.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.485+0000 7ff22c53d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2241449c0 con 0x7ff22413c480 2024-03-25T18:32:13.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.485+0000 7ff22c53d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff224144ca0 con 0x7ff2240b4670 2024-03-25T18:32:13.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.485+0000 7ff22c53d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff22413f2b0 con 0x7ff2240b4ff0 2024-03-25T18:32:13.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.485+0000 7ff22c53d640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:13.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.485+0000 7ff22c53d640 10 monclient(hunting): authenticate will time out at 7787.745399s 2024-03-25T18:32:13.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.501+0000 7ff229ad8640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7ff2240b4ff0 0x7ff224145f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:48978/0 (socket says 172.21.15.3:48978) 2024-03-25T18:32:13.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.501+0000 7ff229ad8640 1 -- 172.21.15.3:0/2190927732 learned_addr learned my addr 172.21.15.3:0/2190927732 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:32:13.507 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.506+0000 7ff21affd640 10 client.?.objecter ms_handle_connect 0x7ff22413c480 2024-03-25T18:32:13.507 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.506+0000 7ff21affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:13.508 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.507+0000 7ff21affd640 10 client.?.objecter ms_handle_connect 0x7ff2240b4ff0 2024-03-25T18:32:13.508 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.507+0000 7ff21affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:13.511 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.510+0000 7ff21affd640 10 client.?.objecter ms_handle_connect 0x7ff2240b4670 2024-03-25T18:32:13.511 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.510+0000 7ff21affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:13.516 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.515+0000 7ff21affd640 1 -- 172.21.15.3:0/2190927732 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 560676100 0 0) 0x7ff22413f2b0 con 0x7ff2240b4ff0 2024-03-25T18:32:13.516 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.515+0000 7ff21affd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:13.516 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.515+0000 7ff21affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:13.516 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.515+0000 7ff21affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:13.516 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.515+0000 7ff21affd640 10 monclient(hunting): my global_id is 4514 2024-03-25T18:32:13.516 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.515+0000 7ff21affd640 1 -- 172.21.15.3:0/2190927732 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff208004eb0 con 0x7ff2240b4ff0 2024-03-25T18:32:13.519 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.518+0000 7ff21affd640 1 -- 172.21.15.3:0/2190927732 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2355304918 0 0) 0x7ff2241449c0 con 0x7ff22413c480 2024-03-25T18:32:13.520 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.518+0000 7ff21affd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:13.520 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.518+0000 7ff21affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:13.520 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.518+0000 7ff21affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:13.520 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.518+0000 7ff21affd640 10 monclient(hunting): my global_id is 4557 2024-03-25T18:32:13.520 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.518+0000 7ff21affd640 1 -- 172.21.15.3:0/2190927732 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff22413f2b0 con 0x7ff22413c480 2024-03-25T18:32:13.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.520+0000 7ff21affd640 1 -- 172.21.15.3:0/2190927732 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2523205784 0 0) 0x7ff224144ca0 con 0x7ff2240b4670 2024-03-25T18:32:13.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.520+0000 7ff21affd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:13.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.520+0000 7ff21affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:13.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.520+0000 7ff21affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:13.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.520+0000 7ff21affd640 10 monclient(hunting): my global_id is 4510 2024-03-25T18:32:13.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.520+0000 7ff21affd640 1 -- 172.21.15.3:0/2190927732 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2241449c0 con 0x7ff2240b4670 2024-03-25T18:32:13.527 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.525+0000 7ff21affd640 1 -- 172.21.15.3:0/2190927732 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1184912410 0 0) 0x7ff22413f2b0 con 0x7ff22413c480 2024-03-25T18:32:13.527 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.526+0000 7ff21affd640 1 -- 172.21.15.3:0/2190927732 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff224144ca0 con 0x7ff22413c480 2024-03-25T18:32:13.527 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.526+0000 7ff21affd640 1 -- 172.21.15.3:0/2190927732 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7ff21c001b00 con 0x7ff22413c480 2024-03-25T18:32:13.527 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.526+0000 7ff21affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:13.527 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.526+0000 7ff21affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:13.527 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.526+0000 7ff21affd640 10 monclient(hunting): dump: 2024-03-25T18:32:13.527 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:13.528 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:13.528 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:13.528 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:13.528 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:13.528 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:13.528 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:13.528 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:13.528 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:13.528 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:13.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.526+0000 7ff21affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:13.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.526+0000 7ff21affd640 1 -- 172.21.15.3:0/2190927732 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3516991726 0 0) 0x7ff2241449c0 con 0x7ff2240b4670 2024-03-25T18:32:13.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.526+0000 7ff21affd640 1 -- 172.21.15.3:0/2190927732 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff22413f2b0 con 0x7ff2240b4670 2024-03-25T18:32:13.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.526+0000 7ff21affd640 1 -- 172.21.15.3:0/2190927732 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7ff220001930 con 0x7ff2240b4670 2024-03-25T18:32:13.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.526+0000 7ff21affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:13.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.526+0000 7ff21affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:13.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.526+0000 7ff21affd640 10 monclient(hunting): dump: 2024-03-25T18:32:13.529 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:13.529 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:13.529 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:13.529 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:13.529 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:13.529 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:13.529 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:13.529 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:13.529 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:13.529 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:13.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.532+0000 7ff21affd640 1 -- 172.21.15.3:0/2190927732 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1433340813 0 0) 0x7ff22413f2b0 con 0x7ff2240b4670 2024-03-25T18:32:13.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.532+0000 7ff21affd640 1 -- 172.21.15.3:0/2190927732 >> v1:172.21.15.3:6790/0 conn(0x7ff2240b4ff0 legacy=0x7ff224145f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:13.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.532+0000 7ff21affd640 1 -- 172.21.15.3:0/2190927732 >> v1:172.21.15.3:6789/0 conn(0x7ff22413c480 legacy=0x7ff2241e27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:13.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.532+0000 7ff21affd640 10 monclient: _finish_hunting 0 2024-03-25T18:32:13.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.532+0000 7ff21affd640 1 monclient: found mon.b 2024-03-25T18:32:13.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.532+0000 7ff21affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:13.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.532+0000 7ff21affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:32:13.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.532+0000 7ff21affd640 1 -- 172.21.15.3:0/2190927732 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff224143e00 con 0x7ff2240b4670 2024-03-25T18:32:13.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.532+0000 7ff21affd640 10 monclient: _finish_auth 0 2024-03-25T18:32:13.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.532+0000 7ff21affd640 10 monclient: _check_auth_tickets 2024-03-25T18:32:13.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.532+0000 7ff21affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:13.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.532+0000 7ff22c53d640 5 monclient: authenticate success, global_id 4510 2024-03-25T18:32:13.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.532+0000 7ff22c53d640 10 monclient: _renew_subs 2024-03-25T18:32:13.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.532+0000 7ff22c53d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:32:13.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.532+0000 7ff22c53d640 1 -- 172.21.15.3:0/2190927732 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff224146640 con 0x7ff2240b4670 2024-03-25T18:32:13.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.532+0000 7ff22c53d640 10 client.4510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:13.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.532+0000 7ff22c53d640 10 monclient: _renew_subs 2024-03-25T18:32:13.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.532+0000 7ff22c53d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:32:13.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.532+0000 7ff22c53d640 1 -- 172.21.15.3:0/2190927732 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff22413cad0 con 0x7ff2240b4670 2024-03-25T18:32:13.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.532+0000 7ff22c53d640 1 librados: init done 2024-03-25T18:32:13.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.533+0000 7ff22c53d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:13.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.533+0000 7ff22c53d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:13.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.533+0000 7ff22c53d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:32:13.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.533+0000 7ff22c53d640 1 -- 172.21.15.3:0/2190927732 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff224143f90 con 0x7ff2240b4670 2024-03-25T18:32:13.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.559+0000 7ff21affd640 1 -- 172.21.15.3:0/2190927732 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff220003610 con 0x7ff2240b4670 2024-03-25T18:32:13.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.559+0000 7ff21affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:13.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.559+0000 7ff228ad6640 4 set_mon_vals no callback set 2024-03-25T18:32:13.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.560+0000 7ff21affd640 1 -- 172.21.15.3:0/2190927732 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7ff220001d80 con 0x7ff2240b4670 2024-03-25T18:32:13.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.560+0000 7ff21affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:13.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.560+0000 7ff21affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:13.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.560+0000 7ff21affd640 10 monclient: dump: 2024-03-25T18:32:13.562 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:13.562 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:13.562 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:13.562 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:13.562 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:13.562 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:13.562 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:13.562 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:13.562 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:13.562 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:13.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.560+0000 7ff21affd640 1 -- 172.21.15.3:0/2190927732 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7ff220019850 con 0x7ff2240b4670 2024-03-25T18:32:13.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.561+0000 7ff229ad8640 10 monclient: get_auth_request con 0x7ff20806dfd0 auth_method 0 2024-03-25T18:32:13.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.561+0000 7ff21affd640 10 client.4510.objecter ms_handle_connect 0x7ff20806dfd0 2024-03-25T18:32:13.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.563+0000 7ff21affd640 1 -- 172.21.15.3:0/2190927732 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7ff220053570 con 0x7ff2240b4670 2024-03-25T18:32:13.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.563+0000 7ff21affd640 10 client.4510.objecter ms_dispatch 0x7ff224000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:13.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.563+0000 7ff21affd640 3 client.4510.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:13.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.563+0000 7ff21affd640 3 client.4510.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:13.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.564+0000 7ff21affd640 20 client.4510.objecter dump_active .. 0 homeless 2024-03-25T18:32:13.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.843+0000 7ff21affd640 1 -- 172.21.15.3:0/2190927732 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff22004ef70 con 0x7ff2240b4670 2024-03-25T18:32:13.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.843+0000 7ff21affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:13.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:13.843+0000 7ff21affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:14.084 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:14.083+0000 7ff22c53d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:14.084 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:14.083+0000 7ff22c53d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:14.084 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:14.083+0000 7ff22c53d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:32:14.084 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:14.083+0000 7ff22c53d640 1 -- 172.21.15.3:0/2190927732 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff22406dbd0 con 0x7ff2240b4670 2024-03-25T18:32:14.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:14.285+0000 7ff21affd640 1 -- 172.21.15.3:0/2190927732 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7ff220051070 con 0x7ff2240b4670 2024-03-25T18:32:14.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:14.285+0000 7ff21affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:14.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:14.285+0000 7ff21affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:14.287 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:32:14.287 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:14.289 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:14.287+0000 7ff22c53d640 10 librados: watch_flush enter 2024-03-25T18:32:14.289 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:14.287+0000 7ff22c53d640 10 librados: watch_flush exit 2024-03-25T18:32:14.289 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:14.287+0000 7ff22c53d640 20 client.4510.objecter shutdown clearing up homeless session... 2024-03-25T18:32:14.289 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:14.287+0000 7ff22c53d640 10 client.4510.objecter successfully canceled tick 2024-03-25T18:32:14.289 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:14.287+0000 7ff22c53d640 1 -- 172.21.15.3:0/2190927732 >> v1:172.21.15.3:6800/4195140966 conn(0x7ff20806dfd0 legacy=0x7ff208070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:14.289 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:14.287+0000 7ff22c53d640 10 monclient: shutdown 2024-03-25T18:32:14.289 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:14.287+0000 7ff22c53d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:14.290 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:14.287+0000 7ff22c53d640 1 -- 172.21.15.3:0/2190927732 >> v1:172.21.15.16:6789/0 conn(0x7ff2240b4670 legacy=0x7ff2241436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:14.290 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:14.287+0000 7ff22c53d640 1 -- 172.21.15.3:0/2190927732 shutdown_connections 2024-03-25T18:32:14.290 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:14.287+0000 7ff22c53d640 1 -- 172.21.15.3:0/2190927732 >> 172.21.15.3:0/2190927732 conn(0x7ff2240617a0 msgr2=0x7ff224140780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:14.290 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:14.288+0000 7ff22c53d640 1 -- 172.21.15.3:0/2190927732 shutdown_connections 2024-03-25T18:32:14.290 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:14.288+0000 7ff22c53d640 1 -- 172.21.15.3:0/2190927732 wait complete. 2024-03-25T18:32:14.290 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:14.288+0000 7ff22c53d640 1 librados: shutdown 2024-03-25T18:32:14.299 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:21.106 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:21.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.220+0000 7f509ba9d640 10 monclient: get_monmap_and_config 2024-03-25T18:32:21.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.220+0000 7f509ba9d640 10 monclient: build_initial_monmap 2024-03-25T18:32:21.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.220+0000 7f509ba9d640 10 monclient: monmap: 2024-03-25T18:32:21.222 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:32:21.222 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:21.222 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:32:21.221795+0000 2024-03-25T18:32:21.222 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:32:21.221795+0000 2024-03-25T18:32:21.222 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:21.223 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:21.223 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:32:21.223 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:32:21.223 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:32:21.223 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:21.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.221+0000 7f509ba9d640 1 Processor -- start 2024-03-25T18:32:21.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.221+0000 7f509ba9d640 1 -- start start 2024-03-25T18:32:21.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.221+0000 7f509ba9d640 10 monclient: init 2024-03-25T18:32:21.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.222+0000 7f509ba9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:21.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.222+0000 7f509ba9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:32:21.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.222+0000 7f509ba9d640 10 monclient(hunting): picked mon.noname-c con 0x7f50940b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:32:21.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.222+0000 7f509ba9d640 10 monclient(hunting): picked mon.noname-a con 0x7f50940b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:32:21.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.222+0000 7f509ba9d640 10 monclient(hunting): picked mon.noname-b con 0x7f509413c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:32:21.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.222+0000 7f509ba9d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50940ab0e0 con 0x7f50940b4670 2024-03-25T18:32:21.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.222+0000 7f509ba9d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50940ab3c0 con 0x7f50940b4ff0 2024-03-25T18:32:21.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.222+0000 7f509ba9d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50940a9f30 con 0x7f509413c480 2024-03-25T18:32:21.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.222+0000 7f509ba9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:21.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.222+0000 7f509ba9d640 10 monclient(hunting): authenticate will time out at 7795.483209s 2024-03-25T18:32:21.235 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.234+0000 7f509a03a640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f509413c480 0x7f509414dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:34340/0 (socket says 172.21.15.3:34340) 2024-03-25T18:32:21.235 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.234+0000 7f509a03a640 1 -- 172.21.15.3:0/292566276 learned_addr learned my addr 172.21.15.3:0/292566276 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:32:21.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.245+0000 7f5098837640 1 -- 172.21.15.3:0/292566276 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2541597645 0 0) 0x7f50940a9f30 con 0x7f509413c480 2024-03-25T18:32:21.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.245+0000 7f5098837640 10 monclient(hunting): _negotiate 2024-03-25T18:32:21.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.245+0000 7f5098837640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:21.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.245+0000 7f5098837640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:21.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.245+0000 7f5098837640 10 monclient(hunting): my global_id is 4517 2024-03-25T18:32:21.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.245+0000 7f5098837640 1 -- 172.21.15.3:0/292566276 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5080003520 con 0x7f509413c480 2024-03-25T18:32:21.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.249+0000 7f5098837640 1 -- 172.21.15.3:0/292566276 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 902033712 0 0) 0x7f50940ab0e0 con 0x7f50940b4670 2024-03-25T18:32:21.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.249+0000 7f5098837640 10 monclient(hunting): _negotiate 2024-03-25T18:32:21.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.249+0000 7f5098837640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:21.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.249+0000 7f5098837640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:21.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.249+0000 7f5098837640 10 monclient(hunting): my global_id is 4560 2024-03-25T18:32:21.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.249+0000 7f5098837640 1 -- 172.21.15.3:0/292566276 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50940a9f30 con 0x7f50940b4670 2024-03-25T18:32:21.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.258+0000 7f5098837640 1 -- 172.21.15.3:0/292566276 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4171511595 0 0) 0x7f50940ab3c0 con 0x7f50940b4ff0 2024-03-25T18:32:21.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.258+0000 7f5098837640 10 monclient(hunting): _negotiate 2024-03-25T18:32:21.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.258+0000 7f5098837640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:21.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.258+0000 7f5098837640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:21.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.258+0000 7f5098837640 10 monclient(hunting): my global_id is 4513 2024-03-25T18:32:21.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.258+0000 7f5098837640 1 -- 172.21.15.3:0/292566276 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50940ab0e0 con 0x7f50940b4ff0 2024-03-25T18:32:21.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.262+0000 7f5098837640 1 -- 172.21.15.3:0/292566276 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3280225863 0 0) 0x7f50940a9f30 con 0x7f50940b4670 2024-03-25T18:32:21.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.262+0000 7f5098837640 1 -- 172.21.15.3:0/292566276 >> v1:172.21.15.3:6790/0 conn(0x7f509413c480 legacy=0x7f509414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:21.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.262+0000 7f5098837640 1 -- 172.21.15.3:0/292566276 >> v1:172.21.15.16:6789/0 conn(0x7f50940b4ff0 legacy=0x7f509413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:21.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.262+0000 7f5098837640 10 monclient: _finish_hunting 0 2024-03-25T18:32:21.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.262+0000 7f5098837640 1 monclient: found mon.noname-a 2024-03-25T18:32:21.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.262+0000 7f5098837640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:21.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.262+0000 7f5098837640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:32:21.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.262+0000 7f5098837640 1 -- 172.21.15.3:0/292566276 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50940a9b70 con 0x7f50940b4670 2024-03-25T18:32:21.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.262+0000 7f5098837640 10 monclient: _finish_auth 0 2024-03-25T18:32:21.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.262+0000 7f5098837640 10 monclient: _check_auth_tickets 2024-03-25T18:32:21.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.262+0000 7f5098837640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:21.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.262+0000 7f5098837640 1 -- 172.21.15.3:0/292566276 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f5084001670 con 0x7f50940b4670 2024-03-25T18:32:21.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.262+0000 7f5098837640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:21.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.262+0000 7f5098837640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:21.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.262+0000 7f5098837640 10 monclient: dump: 2024-03-25T18:32:21.265 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:21.265 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:21.265 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:21.265 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:21.265 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:21.265 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:21.265 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:21.265 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:21.265 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:21.265 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:21.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.262+0000 7f509ba9d640 5 monclient: authenticate success, global_id 4560 2024-03-25T18:32:21.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.262+0000 7f509ba9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:21.273 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.272+0000 7f5098837640 1 -- 172.21.15.3:0/292566276 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50840017d0 con 0x7f50940b4670 2024-03-25T18:32:21.273 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.272+0000 7f5098837640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:21.273 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.272+0000 7f509ba9d640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:21.273 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.272+0000 7f509ba9d640 4 set_mon_vals no callback set 2024-03-25T18:32:21.274 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.272+0000 7f509ba9d640 10 monclient: shutdown 2024-03-25T18:32:21.274 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.272+0000 7f509ba9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:21.274 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.272+0000 7f509ba9d640 1 -- 172.21.15.3:0/292566276 >> v1:172.21.15.3:6789/0 conn(0x7f50940b4670 legacy=0x7f50941425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:21.274 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.273+0000 7f509ba9d640 1 -- 172.21.15.3:0/292566276 shutdown_connections 2024-03-25T18:32:21.275 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.273+0000 7f509ba9d640 1 -- 172.21.15.3:0/292566276 >> 172.21.15.3:0/292566276 conn(0x7f50940617a0 msgr2=0x7f5094061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:21.275 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.273+0000 7f509ba9d640 1 -- 172.21.15.3:0/292566276 shutdown_connections 2024-03-25T18:32:21.275 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.273+0000 7f509ba9d640 1 -- 172.21.15.3:0/292566276 wait complete. 2024-03-25T18:32:21.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.274+0000 7f509ba9d640 10 monclient: build_initial_monmap 2024-03-25T18:32:21.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.274+0000 7f509ba9d640 10 monclient: monmap: 2024-03-25T18:32:21.276 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:32:21.276 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:21.276 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:32:21.276 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:32:21.277 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:21.277 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:32:21.277 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:32:21.277 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:32:21.277 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:32:21.277 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:21.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.274+0000 7f509ba9d640 1 librados: starting msgr at 2024-03-25T18:32:21.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.274+0000 7f509ba9d640 1 librados: starting objecter 2024-03-25T18:32:21.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.274+0000 7f509ba9d640 1 Processor -- start 2024-03-25T18:32:21.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.274+0000 7f509ba9d640 1 -- start start 2024-03-25T18:32:21.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.274+0000 7f509ba9d640 1 librados: setting wanted keys 2024-03-25T18:32:21.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.274+0000 7f509ba9d640 1 librados: calling monclient init 2024-03-25T18:32:21.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.274+0000 7f509ba9d640 10 monclient: init 2024-03-25T18:32:21.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.275+0000 7f509ba9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:21.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.275+0000 7f509ba9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:32:21.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.275+0000 7f509ba9d640 10 monclient(hunting): picked mon.noname-b con 0x7f50940b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:32:21.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.275+0000 7f509ba9d640 10 monclient(hunting): picked mon.noname-c con 0x7f50940b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:32:21.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.275+0000 7f509ba9d640 10 monclient(hunting): picked mon.noname-a con 0x7f509413c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:32:21.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.275+0000 7f509ba9d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5094145f00 con 0x7f509413c480 2024-03-25T18:32:21.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.275+0000 7f509ba9d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50941dbf50 con 0x7f50940b4670 2024-03-25T18:32:21.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.275+0000 7f509ba9d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f509413d9d0 con 0x7f50940b4ff0 2024-03-25T18:32:21.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.275+0000 7f509ba9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:21.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.275+0000 7f509ba9d640 10 monclient(hunting): authenticate will time out at 7795.535747s 2024-03-25T18:32:21.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.277+0000 7f5099839640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f50940b4670 0x7f509413e0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:54198/0 (socket says 172.21.15.3:54198) 2024-03-25T18:32:21.279 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.277+0000 7f5099839640 1 -- 172.21.15.3:0/944749180 learned_addr learned my addr 172.21.15.3:0/944749180 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:32:21.279 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.277+0000 7f508a7fc640 10 client.?.objecter ms_handle_connect 0x7f50940b4670 2024-03-25T18:32:21.279 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.277+0000 7f508a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:21.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.284+0000 7f508a7fc640 1 -- 172.21.15.3:0/944749180 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2844535087 0 0) 0x7f50941dbf50 con 0x7f50940b4670 2024-03-25T18:32:21.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.284+0000 7f508a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:21.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.284+0000 7f508a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:21.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.284+0000 7f508a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:21.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.284+0000 7f508a7fc640 10 monclient(hunting): my global_id is 4516 2024-03-25T18:32:21.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.284+0000 7f508a7fc640 1 -- 172.21.15.3:0/944749180 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5078004eb0 con 0x7f50940b4670 2024-03-25T18:32:21.297 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.295+0000 7f508a7fc640 1 -- 172.21.15.3:0/944749180 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1665167538 0 0) 0x7f5078004eb0 con 0x7f50940b4670 2024-03-25T18:32:21.297 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.296+0000 7f508a7fc640 1 -- 172.21.15.3:0/944749180 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50941dbf50 con 0x7f50940b4670 2024-03-25T18:32:21.297 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.296+0000 7f508a7fc640 1 -- 172.21.15.3:0/944749180 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f5090001b90 con 0x7f50940b4670 2024-03-25T18:32:21.297 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.296+0000 7f508a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:21.297 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.296+0000 7f508a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:21.297 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.296+0000 7f508a7fc640 10 monclient(hunting): dump: 2024-03-25T18:32:21.297 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:21.298 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:21.298 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:21.298 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:21.298 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:21.298 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:21.298 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:21.298 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:21.298 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:21.298 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:21.298 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.296+0000 7f508a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:21.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.302+0000 7f508a7fc640 10 client.?.objecter ms_handle_connect 0x7f50940b4ff0 2024-03-25T18:32:21.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.302+0000 7f508a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:21.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.306+0000 7f508a7fc640 10 client.?.objecter ms_handle_connect 0x7f509413c480 2024-03-25T18:32:21.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.306+0000 7f508a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:21.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.317+0000 7f508a7fc640 1 -- 172.21.15.3:0/944749180 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3896077181 0 0) 0x7f5094145f00 con 0x7f509413c480 2024-03-25T18:32:21.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.317+0000 7f508a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:21.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.317+0000 7f508a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:21.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.317+0000 7f508a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:21.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.317+0000 7f508a7fc640 10 monclient(hunting): my global_id is 4563 2024-03-25T18:32:21.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.317+0000 7f508a7fc640 1 -- 172.21.15.3:0/944749180 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5078004eb0 con 0x7f509413c480 2024-03-25T18:32:21.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.317+0000 7f508a7fc640 1 -- 172.21.15.3:0/944749180 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2356858240 0 0) 0x7f50941dbf50 con 0x7f50940b4670 2024-03-25T18:32:21.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.317+0000 7f508a7fc640 1 -- 172.21.15.3:0/944749180 >> v1:172.21.15.3:6790/0 conn(0x7f50940b4ff0 legacy=0x7f50941db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:21.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.317+0000 7f508a7fc640 1 -- 172.21.15.3:0/944749180 >> v1:172.21.15.3:6789/0 conn(0x7f509413c480 legacy=0x7f50941db840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:21.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.317+0000 7f508a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:32:21.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.317+0000 7f508a7fc640 1 monclient: found mon.b 2024-03-25T18:32:21.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.317+0000 7f508a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:21.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.317+0000 7f508a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:32:21.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.317+0000 7f508a7fc640 1 -- 172.21.15.3:0/944749180 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f509413dcb0 con 0x7f50940b4670 2024-03-25T18:32:21.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.317+0000 7f508a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:32:21.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.317+0000 7f508a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:32:21.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.317+0000 7f508a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:21.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.317+0000 7f509ba9d640 5 monclient: authenticate success, global_id 4516 2024-03-25T18:32:21.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.317+0000 7f509ba9d640 10 monclient: _renew_subs 2024-03-25T18:32:21.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.317+0000 7f509ba9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:32:21.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.317+0000 7f509ba9d640 1 -- 172.21.15.3:0/944749180 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f509413c930 con 0x7f50940b4670 2024-03-25T18:32:21.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.317+0000 7f509ba9d640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:21.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.317+0000 7f509ba9d640 10 monclient: _renew_subs 2024-03-25T18:32:21.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.317+0000 7f509ba9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:32:21.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.317+0000 7f509ba9d640 1 -- 172.21.15.3:0/944749180 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f509413f7e0 con 0x7f50940b4670 2024-03-25T18:32:21.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.317+0000 7f509ba9d640 1 librados: init done 2024-03-25T18:32:21.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.318+0000 7f509ba9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:21.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.318+0000 7f509ba9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:21.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.318+0000 7f509ba9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:32:21.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.318+0000 7f509ba9d640 1 -- 172.21.15.3:0/944749180 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f509406dbd0 con 0x7f50940b4670 2024-03-25T18:32:21.603 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.602+0000 7f508a7fc640 1 -- 172.21.15.3:0/944749180 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5090001cf0 con 0x7f50940b4670 2024-03-25T18:32:21.603 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.602+0000 7f508a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:21.603 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.602+0000 7f508a7fc640 1 -- 172.21.15.3:0/944749180 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f5090003ae0 con 0x7f50940b4670 2024-03-25T18:32:21.603 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.602+0000 7f508a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:21.603 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.602+0000 7f508a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:21.603 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.602+0000 7f508a7fc640 10 monclient: dump: 2024-03-25T18:32:21.604 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:21.604 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:21.604 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:21.604 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:21.604 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:21.604 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:21.604 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:21.604 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:21.604 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:21.604 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:21.604 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.602+0000 7f508bfff640 4 set_mon_vals no callback set 2024-03-25T18:32:21.605 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.603+0000 7f508a7fc640 1 -- 172.21.15.3:0/944749180 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f5090003d00 con 0x7f50940b4670 2024-03-25T18:32:21.605 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.603+0000 7f5099038640 10 monclient: get_auth_request con 0x7f507806dd30 auth_method 0 2024-03-25T18:32:21.605 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.603+0000 7f508a7fc640 10 client.4516.objecter ms_handle_connect 0x7f507806dd30 2024-03-25T18:32:21.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.609+0000 7f508a7fc640 1 -- 172.21.15.3:0/944749180 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f50900549d0 con 0x7f50940b4670 2024-03-25T18:32:21.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.609+0000 7f508a7fc640 10 client.4516.objecter ms_dispatch 0x7f5094000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:21.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.609+0000 7f508a7fc640 3 client.4516.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:21.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.609+0000 7f508a7fc640 3 client.4516.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:21.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.609+0000 7f508a7fc640 20 client.4516.objecter dump_active .. 0 homeless 2024-03-25T18:32:21.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.609+0000 7f508a7fc640 1 -- 172.21.15.3:0/944749180 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5090050310 con 0x7f50940b4670 2024-03-25T18:32:21.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.609+0000 7f508a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:21.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.609+0000 7f508a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:21.864 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.862+0000 7f509ba9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:21.864 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.862+0000 7f509ba9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:21.864 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.862+0000 7f509ba9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:32:21.864 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.862+0000 7f509ba9d640 1 -- 172.21.15.3:0/944749180 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f509413cac0 con 0x7f50940b4670 2024-03-25T18:32:21.978 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.977+0000 7f508a7fc640 1 -- 172.21.15.3:0/944749180 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f5090052410 con 0x7f50940b4670 2024-03-25T18:32:21.978 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.977+0000 7f508a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:21.978 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.977+0000 7f508a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:21.978 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:32:21.978 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:21.981 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.979+0000 7f509ba9d640 10 librados: watch_flush enter 2024-03-25T18:32:21.981 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.979+0000 7f509ba9d640 10 librados: watch_flush exit 2024-03-25T18:32:21.981 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.979+0000 7f509ba9d640 20 client.4516.objecter shutdown clearing up homeless session... 2024-03-25T18:32:21.981 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.979+0000 7f509ba9d640 10 client.4516.objecter successfully canceled tick 2024-03-25T18:32:21.981 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.979+0000 7f509ba9d640 1 -- 172.21.15.3:0/944749180 >> v1:172.21.15.3:6800/4195140966 conn(0x7f507806dd30 legacy=0x7f50780701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:21.981 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.979+0000 7f509ba9d640 10 monclient: shutdown 2024-03-25T18:32:21.982 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.979+0000 7f509ba9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:21.982 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.979+0000 7f509ba9d640 1 -- 172.21.15.3:0/944749180 >> v1:172.21.15.16:6789/0 conn(0x7f50940b4670 legacy=0x7f509413e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:21.982 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.979+0000 7f509ba9d640 1 -- 172.21.15.3:0/944749180 shutdown_connections 2024-03-25T18:32:21.982 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.979+0000 7f509ba9d640 1 -- 172.21.15.3:0/944749180 >> 172.21.15.3:0/944749180 conn(0x7f50940617a0 msgr2=0x7f5094140e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:21.982 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.980+0000 7f509ba9d640 1 -- 172.21.15.3:0/944749180 shutdown_connections 2024-03-25T18:32:21.982 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.980+0000 7f509ba9d640 1 -- 172.21.15.3:0/944749180 wait complete. 2024-03-25T18:32:21.982 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:21.980+0000 7f509ba9d640 1 librados: shutdown 2024-03-25T18:32:21.992 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:22.718 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:07:36.447 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:22.722 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:07:36.451 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:22.765 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:07:36.494 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:22.788 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:07:36.517 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:22.793 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:07:36.485 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:22.798 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:07:36.527 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:22.798 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:07:36.490 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:22.821 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:07:36.513 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:22.827 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:07:36.556 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:28.799 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:28.899 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.898+0000 7f83679e6640 10 monclient: get_monmap_and_config 2024-03-25T18:32:28.899 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.898+0000 7f83679e6640 10 monclient: build_initial_monmap 2024-03-25T18:32:28.899 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.898+0000 7f83679e6640 10 monclient: monmap: 2024-03-25T18:32:28.899 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:32:28.899 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:28.899 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:32:28.899252+0000 2024-03-25T18:32:28.899 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:32:28.899252+0000 2024-03-25T18:32:28.899 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:28.900 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:28.900 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:32:28.900 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:32:28.900 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:32:28.900 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:28.901 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.898+0000 7f83679e6640 1 Processor -- start 2024-03-25T18:32:28.901 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.899+0000 7f83679e6640 1 -- start start 2024-03-25T18:32:28.901 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.899+0000 7f83679e6640 10 monclient: init 2024-03-25T18:32:28.901 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.899+0000 7f83679e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:28.901 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.899+0000 7f83679e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:32:28.902 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.899+0000 7f83679e6640 10 monclient(hunting): picked mon.noname-b con 0x7f83600b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:32:28.902 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.899+0000 7f83679e6640 10 monclient(hunting): picked mon.noname-a con 0x7f83600b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:32:28.902 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.899+0000 7f83679e6640 10 monclient(hunting): picked mon.noname-c con 0x7f836013c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:32:28.902 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.899+0000 7f83679e6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83600ab0e0 con 0x7f83600b4670 2024-03-25T18:32:28.902 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.899+0000 7f83679e6640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83600ab3c0 con 0x7f836013c480 2024-03-25T18:32:28.902 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.899+0000 7f83679e6640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83600a9f30 con 0x7f83600b4ff0 2024-03-25T18:32:28.902 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.899+0000 7f83679e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:28.902 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.899+0000 7f83679e6640 10 monclient(hunting): authenticate will time out at 7803.160113s 2024-03-25T18:32:28.902 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.900+0000 7f8365f83640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f836013c480 0x7f836014dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:51122/0 (socket says 172.21.15.3:51122) 2024-03-25T18:32:28.902 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.900+0000 7f8365f83640 1 -- 172.21.15.3:0/2292706891 learned_addr learned my addr 172.21.15.3:0/2292706891 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:32:28.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.919+0000 7f8357fff640 1 -- 172.21.15.3:0/2292706891 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1013065788 0 0) 0x7f83600ab3c0 con 0x7f836013c480 2024-03-25T18:32:28.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.919+0000 7f8357fff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:28.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.919+0000 7f8357fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:28.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.919+0000 7f8357fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:28.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.919+0000 7f8357fff640 10 monclient(hunting): my global_id is 4519 2024-03-25T18:32:28.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.919+0000 7f8357fff640 1 -- 172.21.15.3:0/2292706891 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f834c003520 con 0x7f836013c480 2024-03-25T18:32:28.925 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.923+0000 7f8357fff640 1 -- 172.21.15.3:0/2292706891 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3319702788 0 0) 0x7f83600ab0e0 con 0x7f83600b4670 2024-03-25T18:32:28.925 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.923+0000 7f8357fff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:28.925 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.923+0000 7f8357fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:28.925 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.923+0000 7f8357fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:28.925 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.923+0000 7f8357fff640 10 monclient(hunting): my global_id is 4566 2024-03-25T18:32:28.925 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.923+0000 7f8357fff640 1 -- 172.21.15.3:0/2292706891 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83600ab3c0 con 0x7f83600b4670 2024-03-25T18:32:28.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.934+0000 7f8357fff640 1 -- 172.21.15.3:0/2292706891 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 497428946 0 0) 0x7f834c003520 con 0x7f836013c480 2024-03-25T18:32:28.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.934+0000 7f8357fff640 1 -- 172.21.15.3:0/2292706891 >> v1:172.21.15.3:6790/0 conn(0x7f83600b4ff0 legacy=0x7f836013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:28.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.934+0000 7f8357fff640 1 -- 172.21.15.3:0/2292706891 >> v1:172.21.15.3:6789/0 conn(0x7f83600b4670 legacy=0x7f83601425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:28.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.934+0000 7f8357fff640 10 monclient: _finish_hunting 0 2024-03-25T18:32:28.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.934+0000 7f8357fff640 1 monclient: found mon.noname-c 2024-03-25T18:32:28.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.934+0000 7f8357fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:28.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.934+0000 7f8357fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:32:28.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.934+0000 7f8357fff640 1 -- 172.21.15.3:0/2292706891 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83600a9b70 con 0x7f836013c480 2024-03-25T18:32:28.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.934+0000 7f8357fff640 10 monclient: _finish_auth 0 2024-03-25T18:32:28.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.934+0000 7f8357fff640 10 monclient: _check_auth_tickets 2024-03-25T18:32:28.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.934+0000 7f8357fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:28.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.934+0000 7f8357fff640 1 -- 172.21.15.3:0/2292706891 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f8358001a10 con 0x7f836013c480 2024-03-25T18:32:28.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.934+0000 7f8357fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:28.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.934+0000 7f8357fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:32:28.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.934+0000 7f8357fff640 10 monclient: dump: 2024-03-25T18:32:28.937 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:28.937 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:28.937 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:28.937 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:28.937 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:28.937 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:28.937 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:28.937 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:28.937 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:28.937 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:28.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.934+0000 7f83679e6640 5 monclient: authenticate success, global_id 4519 2024-03-25T18:32:28.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.934+0000 7f83679e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:28.945 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.944+0000 7f8357fff640 1 -- 172.21.15.3:0/2292706891 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8358001b70 con 0x7f836013c480 2024-03-25T18:32:28.945 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.944+0000 7f8357fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:28.945 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.944+0000 7f8357fff640 1 -- 172.21.15.3:0/2292706891 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f8358001f80 con 0x7f836013c480 2024-03-25T18:32:28.945 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.944+0000 7f8357fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:28.945 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.944+0000 7f8357fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:28.945 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.944+0000 7f8357fff640 10 monclient: dump: 2024-03-25T18:32:28.946 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:28.946 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:28.946 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:28.946 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:28.946 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:28.946 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:28.946 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:28.946 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:28.946 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:28.946 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:28.946 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.944+0000 7f83679e6640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:28.946 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.944+0000 7f83679e6640 4 set_mon_vals no callback set 2024-03-25T18:32:28.948 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.945+0000 7f83679e6640 10 monclient: shutdown 2024-03-25T18:32:28.948 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.945+0000 7f83679e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:28.948 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.945+0000 7f83679e6640 1 -- 172.21.15.3:0/2292706891 >> v1:172.21.15.16:6789/0 conn(0x7f836013c480 legacy=0x7f836014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:28.948 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.945+0000 7f83679e6640 1 -- 172.21.15.3:0/2292706891 shutdown_connections 2024-03-25T18:32:28.948 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.945+0000 7f83679e6640 1 -- 172.21.15.3:0/2292706891 >> 172.21.15.3:0/2292706891 conn(0x7f83600617a0 msgr2=0x7f8360061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:28.948 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.945+0000 7f83679e6640 1 -- 172.21.15.3:0/2292706891 shutdown_connections 2024-03-25T18:32:28.948 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.945+0000 7f83679e6640 1 -- 172.21.15.3:0/2292706891 wait complete. 2024-03-25T18:32:28.948 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.946+0000 7f83679e6640 10 monclient: build_initial_monmap 2024-03-25T18:32:28.948 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.946+0000 7f83679e6640 10 monclient: monmap: 2024-03-25T18:32:28.948 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:32:28.948 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:28.948 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:32:28.948 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:32:28.949 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:28.949 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:32:28.949 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:32:28.949 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:32:28.949 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:32:28.949 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:28.949 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.946+0000 7f83679e6640 1 librados: starting msgr at 2024-03-25T18:32:28.949 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.946+0000 7f83679e6640 1 librados: starting objecter 2024-03-25T18:32:28.949 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.946+0000 7f83679e6640 1 Processor -- start 2024-03-25T18:32:28.949 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.946+0000 7f83679e6640 1 -- start start 2024-03-25T18:32:28.949 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.946+0000 7f83679e6640 1 librados: setting wanted keys 2024-03-25T18:32:28.949 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.946+0000 7f83679e6640 1 librados: calling monclient init 2024-03-25T18:32:28.949 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.946+0000 7f83679e6640 10 monclient: init 2024-03-25T18:32:28.950 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.947+0000 7f83679e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:28.950 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.947+0000 7f83679e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:32:28.950 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.947+0000 7f83679e6640 10 monclient(hunting): picked mon.noname-c con 0x7f83600b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:32:28.950 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.947+0000 7f83679e6640 10 monclient(hunting): picked mon.noname-b con 0x7f83600b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:32:28.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.947+0000 7f83679e6640 10 monclient(hunting): picked mon.noname-a con 0x7f836013c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:32:28.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.947+0000 7f83679e6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83600b0830 con 0x7f836013c480 2024-03-25T18:32:28.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.947+0000 7f83679e6640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83600b0b10 con 0x7f83600b4ff0 2024-03-25T18:32:28.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.947+0000 7f83679e6640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83600b1550 con 0x7f83600b4670 2024-03-25T18:32:28.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.947+0000 7f83679e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:28.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.947+0000 7f83679e6640 10 monclient(hunting): authenticate will time out at 7803.208102s 2024-03-25T18:32:28.967 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.966+0000 7f8365f83640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f836013c480 0x7f83601e2380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55794/0 (socket says 172.21.15.3:55794) 2024-03-25T18:32:28.967 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.966+0000 7f8365782640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f83600b4670 0x7f83600ae720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:53588/0 (socket says 172.21.15.3:53588) 2024-03-25T18:32:28.967 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.966+0000 7f8365f83640 1 -- 172.21.15.3:0/3782082462 learned_addr learned my addr 172.21.15.3:0/3782082462 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:32:28.968 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.967+0000 7f8355ffb640 10 client.?.objecter ms_handle_connect 0x7f836013c480 2024-03-25T18:32:28.968 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.967+0000 7f8355ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:28.969 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.968+0000 7f8355ffb640 10 client.?.objecter ms_handle_connect 0x7f83600b4670 2024-03-25T18:32:28.969 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.968+0000 7f8355ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:28.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.976+0000 7f8355ffb640 10 client.?.objecter ms_handle_connect 0x7f83600b4ff0 2024-03-25T18:32:28.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.976+0000 7f8355ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:28.979 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.978+0000 7f8355ffb640 1 -- 172.21.15.3:0/3782082462 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 385822618 0 0) 0x7f83600b1550 con 0x7f83600b4670 2024-03-25T18:32:28.979 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.978+0000 7f8355ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:28.979 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.978+0000 7f8355ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:28.979 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.978+0000 7f8355ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:28.980 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.978+0000 7f8355ffb640 10 monclient(hunting): my global_id is 4526 2024-03-25T18:32:28.980 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.978+0000 7f8355ffb640 1 -- 172.21.15.3:0/3782082462 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8344004eb0 con 0x7f83600b4670 2024-03-25T18:32:28.990 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.989+0000 7f8355ffb640 1 -- 172.21.15.3:0/3782082462 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1732835109 0 0) 0x7f83600b0b10 con 0x7f83600b4ff0 2024-03-25T18:32:28.990 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.989+0000 7f8355ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:28.991 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.989+0000 7f8355ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:28.991 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.989+0000 7f8355ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:28.991 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.989+0000 7f8355ffb640 10 monclient(hunting): my global_id is 4522 2024-03-25T18:32:28.991 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.989+0000 7f8355ffb640 1 -- 172.21.15.3:0/3782082462 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83600b1550 con 0x7f83600b4ff0 2024-03-25T18:32:28.995 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.994+0000 7f8355ffb640 1 -- 172.21.15.3:0/3782082462 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2435561875 0 0) 0x7f83600b0830 con 0x7f836013c480 2024-03-25T18:32:28.996 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.994+0000 7f8355ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:28.996 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.994+0000 7f8355ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:28.996 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.994+0000 7f8355ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:28.996 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.994+0000 7f8355ffb640 10 monclient(hunting): my global_id is 4569 2024-03-25T18:32:28.996 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:28.994+0000 7f8355ffb640 1 -- 172.21.15.3:0/3782082462 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83600b0b10 con 0x7f836013c480 2024-03-25T18:32:29.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.005+0000 7f8355ffb640 1 -- 172.21.15.3:0/3782082462 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1597694973 0 0) 0x7f83600b0b10 con 0x7f836013c480 2024-03-25T18:32:29.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.005+0000 7f8355ffb640 1 -- 172.21.15.3:0/3782082462 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83600b0830 con 0x7f836013c480 2024-03-25T18:32:29.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.005+0000 7f8355ffb640 1 -- 172.21.15.3:0/3782082462 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f8358001e80 con 0x7f836013c480 2024-03-25T18:32:29.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.005+0000 7f8355ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:29.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.005+0000 7f8355ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:29.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.005+0000 7f8355ffb640 10 monclient(hunting): dump: 2024-03-25T18:32:29.007 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:29.007 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:29.007 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:29.007 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:29.007 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:29.007 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:29.007 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:29.008 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:29.008 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:29.008 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:29.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.005+0000 7f8355ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:29.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.007+0000 7f8355ffb640 1 -- 172.21.15.3:0/3782082462 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1196879426 0 0) 0x7f8344004eb0 con 0x7f83600b4670 2024-03-25T18:32:29.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.007+0000 7f8355ffb640 1 -- 172.21.15.3:0/3782082462 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83600b0b10 con 0x7f83600b4670 2024-03-25T18:32:29.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.007+0000 7f8355ffb640 1 -- 172.21.15.3:0/3782082462 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f835c001970 con 0x7f83600b4670 2024-03-25T18:32:29.009 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.007+0000 7f8355ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:29.009 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.007+0000 7f8355ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:29.009 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.007+0000 7f8355ffb640 10 monclient(hunting): dump: 2024-03-25T18:32:29.009 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:29.009 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:29.009 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:29.009 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:29.009 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:29.009 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:29.009 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:29.009 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:29.009 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:29.009 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:29.012 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.011+0000 7f8355ffb640 1 -- 172.21.15.3:0/3782082462 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3273989031 0 0) 0x7f83600b1550 con 0x7f83600b4ff0 2024-03-25T18:32:29.012 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.011+0000 7f8355ffb640 1 -- 172.21.15.3:0/3782082462 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8344004eb0 con 0x7f83600b4ff0 2024-03-25T18:32:29.012 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.011+0000 7f8355ffb640 1 -- 172.21.15.3:0/3782082462 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f83500018b0 con 0x7f83600b4ff0 2024-03-25T18:32:29.012 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.011+0000 7f8355ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:29.012 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.011+0000 7f8355ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:29.012 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.011+0000 7f8355ffb640 10 monclient(hunting): dump: 2024-03-25T18:32:29.012 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:29.012 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:29.012 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:29.012 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:29.012 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:29.012 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:29.013 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:29.013 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:29.013 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:29.013 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:29.017 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.016+0000 7f8355ffb640 1 -- 172.21.15.3:0/3782082462 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2705643220 0 0) 0x7f83600b0b10 con 0x7f83600b4670 2024-03-25T18:32:29.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.016+0000 7f8355ffb640 1 -- 172.21.15.3:0/3782082462 >> v1:172.21.15.16:6789/0 conn(0x7f83600b4ff0 legacy=0x7f83600aef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:29.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.016+0000 7f8355ffb640 1 -- 172.21.15.3:0/3782082462 >> v1:172.21.15.3:6789/0 conn(0x7f836013c480 legacy=0x7f83601e2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:29.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.016+0000 7f8355ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:32:29.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.016+0000 7f8355ffb640 1 monclient: found mon.c 2024-03-25T18:32:29.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.016+0000 7f8355ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:29.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.016+0000 7f8355ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:32:29.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.016+0000 7f8355ffb640 1 -- 172.21.15.3:0/3782082462 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83600b0f30 con 0x7f83600b4670 2024-03-25T18:32:29.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.016+0000 7f8355ffb640 10 monclient: _finish_auth 0 2024-03-25T18:32:29.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.016+0000 7f8355ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:32:29.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.016+0000 7f8355ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:29.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.016+0000 7f83679e6640 5 monclient: authenticate success, global_id 4526 2024-03-25T18:32:29.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.016+0000 7f83679e6640 10 monclient: _renew_subs 2024-03-25T18:32:29.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.016+0000 7f83679e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:32:29.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.016+0000 7f83679e6640 1 -- 172.21.15.3:0/3782082462 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83600b1180 con 0x7f83600b4670 2024-03-25T18:32:29.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.016+0000 7f83679e6640 10 client.4526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:29.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.016+0000 7f83679e6640 10 monclient: _renew_subs 2024-03-25T18:32:29.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.016+0000 7f83679e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:32:29.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.016+0000 7f83679e6640 1 -- 172.21.15.3:0/3782082462 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f83600af840 con 0x7f83600b4670 2024-03-25T18:32:29.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.016+0000 7f83679e6640 1 librados: init done 2024-03-25T18:32:29.020 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.017+0000 7f83679e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:29.020 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.017+0000 7f83679e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:29.020 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.017+0000 7f83679e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:32:29.020 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.017+0000 7f83679e6640 1 -- 172.21.15.3:0/3782082462 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83600b13c0 con 0x7f83600b4670 2024-03-25T18:32:29.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.324+0000 7f8355ffb640 1 -- 172.21.15.3:0/3782082462 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f835c003610 con 0x7f83600b4670 2024-03-25T18:32:29.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.324+0000 7f8355ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:29.326 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.324+0000 7f8355ffb640 1 -- 172.21.15.3:0/3782082462 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f835c003960 con 0x7f83600b4670 2024-03-25T18:32:29.326 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.324+0000 7f8355ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:29.326 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.324+0000 7f8355ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:29.326 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.324+0000 7f8355ffb640 10 monclient: dump: 2024-03-25T18:32:29.326 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:29.326 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:29.326 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:29.326 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:29.326 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:29.326 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:29.326 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:29.326 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:29.326 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:29.326 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:29.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.324+0000 7f83577fe640 4 set_mon_vals no callback set 2024-03-25T18:32:29.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.325+0000 7f8355ffb640 1 -- 172.21.15.3:0/3782082462 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f835c003b80 con 0x7f83600b4670 2024-03-25T18:32:29.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.326+0000 7f8364f81640 10 monclient: get_auth_request con 0x7f834406e160 auth_method 0 2024-03-25T18:32:29.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.326+0000 7f8355ffb640 10 client.4526.objecter ms_handle_connect 0x7f834406e160 2024-03-25T18:32:29.332 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.330+0000 7f8355ffb640 1 -- 172.21.15.3:0/3782082462 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f835c054820 con 0x7f83600b4670 2024-03-25T18:32:29.332 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.330+0000 7f8355ffb640 10 client.4526.objecter ms_dispatch 0x7f8360000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:29.332 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.330+0000 7f8355ffb640 3 client.4526.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:29.332 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.330+0000 7f8355ffb640 3 client.4526.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:29.332 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.330+0000 7f8355ffb640 20 client.4526.objecter dump_active .. 0 homeless 2024-03-25T18:32:29.332 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.330+0000 7f8355ffb640 1 -- 172.21.15.3:0/3782082462 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f835c050330 con 0x7f83600b4670 2024-03-25T18:32:29.332 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.331+0000 7f8355ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:29.332 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.331+0000 7f8355ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:29.593 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.592+0000 7f83679e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:29.593 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.592+0000 7f83679e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:29.593 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.592+0000 7f83679e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:32:29.594 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.592+0000 7f83679e6640 1 -- 172.21.15.3:0/3782082462 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83600b16e0 con 0x7f83600b4670 2024-03-25T18:32:29.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.614+0000 7f8355ffb640 1 -- 172.21.15.3:0/3782082462 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f835c052260 con 0x7f83600b4670 2024-03-25T18:32:29.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.614+0000 7f8355ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:29.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.614+0000 7f8355ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:29.616 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:32:29.616 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:29.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.616+0000 7f83679e6640 10 librados: watch_flush enter 2024-03-25T18:32:29.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.616+0000 7f83679e6640 10 librados: watch_flush exit 2024-03-25T18:32:29.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.616+0000 7f83679e6640 20 client.4526.objecter shutdown clearing up homeless session... 2024-03-25T18:32:29.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.616+0000 7f83679e6640 10 client.4526.objecter successfully canceled tick 2024-03-25T18:32:29.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.616+0000 7f83679e6640 1 -- 172.21.15.3:0/3782082462 >> v1:172.21.15.3:6800/4195140966 conn(0x7f834406e160 legacy=0x7f8344070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:29.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.616+0000 7f83679e6640 10 monclient: shutdown 2024-03-25T18:32:29.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.616+0000 7f83679e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:29.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.616+0000 7f83679e6640 1 -- 172.21.15.3:0/3782082462 >> v1:172.21.15.3:6790/0 conn(0x7f83600b4670 legacy=0x7f83600ae720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:29.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.617+0000 7f83679e6640 1 -- 172.21.15.3:0/3782082462 shutdown_connections 2024-03-25T18:32:29.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.617+0000 7f83679e6640 1 -- 172.21.15.3:0/3782082462 >> 172.21.15.3:0/3782082462 conn(0x7f83600617a0 msgr2=0x7f8360145670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:29.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.617+0000 7f83679e6640 1 -- 172.21.15.3:0/3782082462 shutdown_connections 2024-03-25T18:32:29.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.617+0000 7f83679e6640 1 -- 172.21.15.3:0/3782082462 wait complete. 2024-03-25T18:32:29.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:29.617+0000 7f83679e6640 1 librados: shutdown 2024-03-25T18:32:29.630 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:36.436 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:36.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.552+0000 7f64c1b82640 10 monclient: get_monmap_and_config 2024-03-25T18:32:36.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.552+0000 7f64c1b82640 10 monclient: build_initial_monmap 2024-03-25T18:32:36.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.552+0000 7f64c1b82640 10 monclient: monmap: 2024-03-25T18:32:36.553 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:32:36.553 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:36.553 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:32:36.553308+0000 2024-03-25T18:32:36.553 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:32:36.553308+0000 2024-03-25T18:32:36.553 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:36.554 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:36.554 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:32:36.554 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:32:36.554 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:32:36.554 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:36.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.553+0000 7f64c1b82640 1 Processor -- start 2024-03-25T18:32:36.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.553+0000 7f64c1b82640 1 -- start start 2024-03-25T18:32:36.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.553+0000 7f64c1b82640 10 monclient: init 2024-03-25T18:32:36.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.553+0000 7f64c1b82640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:36.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.553+0000 7f64c1b82640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:32:36.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.553+0000 7f64c1b82640 10 monclient(hunting): picked mon.noname-a con 0x7f64bc0b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:32:36.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.553+0000 7f64c1b82640 10 monclient(hunting): picked mon.noname-c con 0x7f64bc0b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:32:36.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.553+0000 7f64c1b82640 10 monclient(hunting): picked mon.noname-b con 0x7f64bc13c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:32:36.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.553+0000 7f64c1b82640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64bc0ab0e0 con 0x7f64bc0b4ff0 2024-03-25T18:32:36.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.553+0000 7f64c1b82640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64bc0ab3c0 con 0x7f64bc0b4670 2024-03-25T18:32:36.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.553+0000 7f64c1b82640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64bc0a9f30 con 0x7f64bc13c480 2024-03-25T18:32:36.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.553+0000 7f64c1b82640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:36.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.553+0000 7f64c1b82640 10 monclient(hunting): authenticate will time out at 7810.814317s 2024-03-25T18:32:36.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.557+0000 7f64bbfff640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f64bc13c480 0x7f64bc14dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:58340/0 (socket says 172.21.15.3:58340) 2024-03-25T18:32:36.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.557+0000 7f64bbfff640 1 -- 172.21.15.3:0/1868310415 learned_addr learned my addr 172.21.15.3:0/1868310415 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:32:36.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.563+0000 7f64ba7fc640 1 -- 172.21.15.3:0/1868310415 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1797264168 0 0) 0x7f64bc0a9f30 con 0x7f64bc13c480 2024-03-25T18:32:36.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.563+0000 7f64ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:36.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.563+0000 7f64ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:36.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.563+0000 7f64ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:36.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.563+0000 7f64ba7fc640 10 monclient(hunting): my global_id is 4529 2024-03-25T18:32:36.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.563+0000 7f64ba7fc640 1 -- 172.21.15.3:0/1868310415 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64a4003520 con 0x7f64bc13c480 2024-03-25T18:32:36.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.568+0000 7f64ba7fc640 1 -- 172.21.15.3:0/1868310415 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1363281307 0 0) 0x7f64bc0ab0e0 con 0x7f64bc0b4ff0 2024-03-25T18:32:36.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.568+0000 7f64ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:36.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.568+0000 7f64ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:36.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.568+0000 7f64ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:36.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.568+0000 7f64ba7fc640 10 monclient(hunting): my global_id is 4572 2024-03-25T18:32:36.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.568+0000 7f64ba7fc640 1 -- 172.21.15.3:0/1868310415 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64bc0a9f30 con 0x7f64bc0b4ff0 2024-03-25T18:32:36.579 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.578+0000 7f64ba7fc640 1 -- 172.21.15.3:0/1868310415 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1262724611 0 0) 0x7f64a4003520 con 0x7f64bc13c480 2024-03-25T18:32:36.580 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.578+0000 7f64ba7fc640 1 -- 172.21.15.3:0/1868310415 >> v1:172.21.15.16:6789/0 conn(0x7f64bc0b4670 legacy=0x7f64bc1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:36.580 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.578+0000 7f64ba7fc640 1 -- 172.21.15.3:0/1868310415 >> v1:172.21.15.3:6789/0 conn(0x7f64bc0b4ff0 legacy=0x7f64bc13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:36.580 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.578+0000 7f64ba7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:32:36.580 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.578+0000 7f64ba7fc640 1 monclient: found mon.noname-b 2024-03-25T18:32:36.580 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.578+0000 7f64ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:36.580 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.578+0000 7f64ba7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-25T18:32:36.580 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.578+0000 7f64ba7fc640 1 -- 172.21.15.3:0/1868310415 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64bc0a9b70 con 0x7f64bc13c480 2024-03-25T18:32:36.580 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.578+0000 7f64ba7fc640 10 monclient: _finish_auth 0 2024-03-25T18:32:36.580 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.578+0000 7f64ba7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:32:36.580 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.578+0000 7f64ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:36.580 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.578+0000 7f64ba7fc640 1 -- 172.21.15.3:0/1868310415 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f64b4001a10 con 0x7f64bc13c480 2024-03-25T18:32:36.580 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.578+0000 7f64ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:36.580 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.578+0000 7f64ba7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:36.580 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.578+0000 7f64ba7fc640 10 monclient: dump: 2024-03-25T18:32:36.580 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:36.580 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:36.581 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:36.581 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:36.581 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:36.581 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:36.581 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:36.581 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:36.581 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:36.581 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:36.581 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.578+0000 7f64c1b82640 5 monclient: authenticate success, global_id 4529 2024-03-25T18:32:36.581 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.578+0000 7f64c1b82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:36.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.586+0000 7f64ba7fc640 1 -- 172.21.15.3:0/1868310415 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64b4001b70 con 0x7f64bc13c480 2024-03-25T18:32:36.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.586+0000 7f64ba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:36.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.586+0000 7f64ba7fc640 1 -- 172.21.15.3:0/1868310415 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f64b4001f80 con 0x7f64bc13c480 2024-03-25T18:32:36.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.586+0000 7f64c1b82640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:36.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.586+0000 7f64c1b82640 4 set_mon_vals no callback set 2024-03-25T18:32:36.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.586+0000 7f64ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:36.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.586+0000 7f64ba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:36.588 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.586+0000 7f64ba7fc640 10 monclient: dump: 2024-03-25T18:32:36.588 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:36.588 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:36.588 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:36.588 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:36.588 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:36.588 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:36.588 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:36.588 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:36.588 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:36.588 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:36.591 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.586+0000 7f64c1b82640 10 monclient: shutdown 2024-03-25T18:32:36.591 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.586+0000 7f64c1b82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:36.591 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.586+0000 7f64c1b82640 1 -- 172.21.15.3:0/1868310415 >> v1:172.21.15.3:6790/0 conn(0x7f64bc13c480 legacy=0x7f64bc14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:36.591 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.586+0000 7f64c1b82640 1 -- 172.21.15.3:0/1868310415 shutdown_connections 2024-03-25T18:32:36.591 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.586+0000 7f64c1b82640 1 -- 172.21.15.3:0/1868310415 >> 172.21.15.3:0/1868310415 conn(0x7f64bc0617a0 msgr2=0x7f64bc061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:36.591 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.587+0000 7f64c1b82640 1 -- 172.21.15.3:0/1868310415 shutdown_connections 2024-03-25T18:32:36.591 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.587+0000 7f64c1b82640 1 -- 172.21.15.3:0/1868310415 wait complete. 2024-03-25T18:32:36.591 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.587+0000 7f64c1b82640 10 monclient: build_initial_monmap 2024-03-25T18:32:36.591 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.587+0000 7f64c1b82640 10 monclient: monmap: 2024-03-25T18:32:36.591 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:32:36.591 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:36.591 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:32:36.591 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:32:36.592 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:36.592 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:32:36.592 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:32:36.592 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:32:36.592 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:32:36.592 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:36.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.587+0000 7f64c1b82640 1 librados: starting msgr at 2024-03-25T18:32:36.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.587+0000 7f64c1b82640 1 librados: starting objecter 2024-03-25T18:32:36.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.588+0000 7f64c1b82640 1 Processor -- start 2024-03-25T18:32:36.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.588+0000 7f64c1b82640 1 -- start start 2024-03-25T18:32:36.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.588+0000 7f64c1b82640 1 librados: setting wanted keys 2024-03-25T18:32:36.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.588+0000 7f64c1b82640 1 librados: calling monclient init 2024-03-25T18:32:36.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.588+0000 7f64c1b82640 10 monclient: init 2024-03-25T18:32:36.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.588+0000 7f64c1b82640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:36.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.588+0000 7f64c1b82640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:32:36.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.589+0000 7f64c1b82640 10 monclient(hunting): picked mon.noname-c con 0x7f64bc0b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:32:36.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.589+0000 7f64c1b82640 10 monclient(hunting): picked mon.noname-b con 0x7f64bc0b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:32:36.593 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.589+0000 7f64c1b82640 10 monclient(hunting): picked mon.noname-a con 0x7f64bc13c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:32:36.593 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.589+0000 7f64c1b82640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64bc0b0be0 con 0x7f64bc13c480 2024-03-25T18:32:36.593 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.589+0000 7f64c1b82640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64bc0ae790 con 0x7f64bc0b4ff0 2024-03-25T18:32:36.593 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.589+0000 7f64c1b82640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64bc14e230 con 0x7f64bc0b4670 2024-03-25T18:32:36.593 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.589+0000 7f64c1b82640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:36.593 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.589+0000 7f64c1b82640 10 monclient(hunting): authenticate will time out at 7810.849509s 2024-03-25T18:32:36.602 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.601+0000 7f64bb7fe640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f64bc0b4670 0x7f64bc0b1370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:58368/0 (socket says 172.21.15.3:58368) 2024-03-25T18:32:36.602 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.601+0000 7f64bb7fe640 1 -- 172.21.15.3:0/3907551109 learned_addr learned my addr 172.21.15.3:0/3907551109 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:32:36.603 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.602+0000 7f64a3fff640 10 client.?.objecter ms_handle_connect 0x7f64bc0b4670 2024-03-25T18:32:36.603 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.602+0000 7f64a3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:36.609 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.608+0000 7f64a3fff640 10 client.?.objecter ms_handle_connect 0x7f64bc0b4ff0 2024-03-25T18:32:36.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.608+0000 7f64a3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:36.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.611+0000 7f64a3fff640 1 -- 172.21.15.3:0/3907551109 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 939218975 0 0) 0x7f64bc14e230 con 0x7f64bc0b4670 2024-03-25T18:32:36.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.611+0000 7f64a3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:36.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.611+0000 7f64a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:36.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.611+0000 7f64a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:36.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.611+0000 7f64a3fff640 10 monclient(hunting): my global_id is 4532 2024-03-25T18:32:36.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.611+0000 7f64a3fff640 1 -- 172.21.15.3:0/3907551109 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64a8004eb0 con 0x7f64bc0b4670 2024-03-25T18:32:36.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.613+0000 7f64a3fff640 10 client.?.objecter ms_handle_connect 0x7f64bc13c480 2024-03-25T18:32:36.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.613+0000 7f64a3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:36.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.617+0000 7f64a3fff640 1 -- 172.21.15.3:0/3907551109 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1604454819 0 0) 0x7f64bc0ae790 con 0x7f64bc0b4ff0 2024-03-25T18:32:36.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.617+0000 7f64a3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:36.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.617+0000 7f64a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:36.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.617+0000 7f64a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:36.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.617+0000 7f64a3fff640 10 monclient(hunting): my global_id is 4528 2024-03-25T18:32:36.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.617+0000 7f64a3fff640 1 -- 172.21.15.3:0/3907551109 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64bc14e230 con 0x7f64bc0b4ff0 2024-03-25T18:32:36.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.628+0000 7f64a3fff640 1 -- 172.21.15.3:0/3907551109 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 197218626 0 0) 0x7f64a8004eb0 con 0x7f64bc0b4670 2024-03-25T18:32:36.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.628+0000 7f64a3fff640 1 -- 172.21.15.3:0/3907551109 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64bc0ae790 con 0x7f64bc0b4670 2024-03-25T18:32:36.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.628+0000 7f64a3fff640 1 -- 172.21.15.3:0/3907551109 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f64ac001a10 con 0x7f64bc0b4670 2024-03-25T18:32:36.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.628+0000 7f64a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:36.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.628+0000 7f64a3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:32:36.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.628+0000 7f64a3fff640 10 monclient(hunting): dump: 2024-03-25T18:32:36.630 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:36.630 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:36.630 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:36.630 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:36.630 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:36.630 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:36.630 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:36.630 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:36.630 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:36.630 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:36.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.628+0000 7f64a3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:36.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.629+0000 7f64a3fff640 1 -- 172.21.15.3:0/3907551109 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407678370 0 0) 0x7f64bc0b0be0 con 0x7f64bc13c480 2024-03-25T18:32:36.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.629+0000 7f64a3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:36.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.629+0000 7f64a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:36.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.629+0000 7f64a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:36.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.629+0000 7f64a3fff640 10 monclient(hunting): my global_id is 4575 2024-03-25T18:32:36.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.629+0000 7f64a3fff640 1 -- 172.21.15.3:0/3907551109 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64a8004eb0 con 0x7f64bc13c480 2024-03-25T18:32:36.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.632+0000 7f64a3fff640 1 -- 172.21.15.3:0/3907551109 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 950677637 0 0) 0x7f64bc14e230 con 0x7f64bc0b4ff0 2024-03-25T18:32:36.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.632+0000 7f64a3fff640 1 -- 172.21.15.3:0/3907551109 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64bc0b0be0 con 0x7f64bc0b4ff0 2024-03-25T18:32:36.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.632+0000 7f64a3fff640 1 -- 172.21.15.3:0/3907551109 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f64b00018b0 con 0x7f64bc0b4ff0 2024-03-25T18:32:36.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.632+0000 7f64a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:36.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.632+0000 7f64a3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:36.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.632+0000 7f64a3fff640 10 monclient(hunting): dump: 2024-03-25T18:32:36.634 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:36.634 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:36.634 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:36.634 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:36.634 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:36.634 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:36.634 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:36.634 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:36.634 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:36.634 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:36.649 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.648+0000 7f64a3fff640 1 -- 172.21.15.3:0/3907551109 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3522999908 0 0) 0x7f64bc0ae790 con 0x7f64bc0b4670 2024-03-25T18:32:36.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.648+0000 7f64a3fff640 1 -- 172.21.15.3:0/3907551109 >> v1:172.21.15.16:6789/0 conn(0x7f64bc0b4ff0 legacy=0x7f64bc0aecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:36.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.648+0000 7f64a3fff640 1 -- 172.21.15.3:0/3907551109 >> v1:172.21.15.3:6789/0 conn(0x7f64bc13c480 legacy=0x7f64bc0af3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:36.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.648+0000 7f64a3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:32:36.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.648+0000 7f64a3fff640 1 monclient: found mon.c 2024-03-25T18:32:36.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.648+0000 7f64a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:36.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.648+0000 7f64a3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:32:36.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.648+0000 7f64a3fff640 1 -- 172.21.15.3:0/3907551109 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64bc14e3c0 con 0x7f64bc0b4670 2024-03-25T18:32:36.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.648+0000 7f64a3fff640 10 monclient: _finish_auth 0 2024-03-25T18:32:36.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.648+0000 7f64a3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:32:36.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.648+0000 7f64a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:36.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.648+0000 7f64c1b82640 5 monclient: authenticate success, global_id 4532 2024-03-25T18:32:36.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.648+0000 7f64c1b82640 10 monclient: _renew_subs 2024-03-25T18:32:36.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.648+0000 7f64c1b82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:32:36.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.648+0000 7f64c1b82640 1 -- 172.21.15.3:0/3907551109 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64bc1e36f0 con 0x7f64bc0b4670 2024-03-25T18:32:36.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.648+0000 7f64c1b82640 10 client.4532.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:36.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.648+0000 7f64c1b82640 10 monclient: _renew_subs 2024-03-25T18:32:36.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.648+0000 7f64c1b82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:32:36.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.648+0000 7f64c1b82640 1 -- 172.21.15.3:0/3907551109 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64bc1e3a60 con 0x7f64bc0b4670 2024-03-25T18:32:36.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.648+0000 7f64c1b82640 1 librados: init done 2024-03-25T18:32:36.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.649+0000 7f64c1b82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:36.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.649+0000 7f64c1b82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:36.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.649+0000 7f64c1b82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:32:36.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.649+0000 7f64c1b82640 1 -- 172.21.15.3:0/3907551109 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64bc146770 con 0x7f64bc0b4670 2024-03-25T18:32:36.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.963+0000 7f64a3fff640 1 -- 172.21.15.3:0/3907551109 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64ac001c50 con 0x7f64bc0b4670 2024-03-25T18:32:36.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.963+0000 7f64a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:36.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.963+0000 7f64a3fff640 1 -- 172.21.15.3:0/3907551109 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f64ac003a60 con 0x7f64bc0b4670 2024-03-25T18:32:36.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.963+0000 7f64a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:36.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.963+0000 7f64a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:36.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.963+0000 7f64a3fff640 10 monclient: dump: 2024-03-25T18:32:36.965 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:36.965 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:36.965 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:36.965 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:36.965 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:36.965 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:36.966 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:36.966 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:36.966 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:36.966 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:36.966 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.963+0000 7f64b9ffb640 4 set_mon_vals no callback set 2024-03-25T18:32:36.966 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.964+0000 7f64a3fff640 1 -- 172.21.15.3:0/3907551109 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f64ac003c80 con 0x7f64bc0b4670 2024-03-25T18:32:36.967 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.964+0000 7f64baffd640 10 monclient: get_auth_request con 0x7f64a806df70 auth_method 0 2024-03-25T18:32:36.967 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.965+0000 7f64a3fff640 10 client.4532.objecter ms_handle_connect 0x7f64a806df70 2024-03-25T18:32:36.971 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.969+0000 7f64a3fff640 1 -- 172.21.15.3:0/3907551109 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f64ac0548b0 con 0x7f64bc0b4670 2024-03-25T18:32:36.971 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.969+0000 7f64a3fff640 10 client.4532.objecter ms_dispatch 0x7f64bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:36.971 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.969+0000 7f64a3fff640 3 client.4532.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:36.971 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.969+0000 7f64a3fff640 3 client.4532.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:36.971 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.970+0000 7f64a3fff640 20 client.4532.objecter dump_active .. 0 homeless 2024-03-25T18:32:36.971 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.970+0000 7f64a3fff640 1 -- 172.21.15.3:0/3907551109 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f64ac0503c0 con 0x7f64bc0b4670 2024-03-25T18:32:36.971 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.970+0000 7f64a3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:36.971 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:36.970+0000 7f64a3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:37.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:37.223+0000 7f64c1b82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:37.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:37.223+0000 7f64c1b82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:37.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:37.223+0000 7f64c1b82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:32:37.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:37.223+0000 7f64c1b82640 1 -- 172.21.15.3:0/3907551109 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64bc0ae920 con 0x7f64bc0b4670 2024-03-25T18:32:37.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:37.261+0000 7f64a3fff640 1 -- 172.21.15.3:0/3907551109 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f64ac0522f0 con 0x7f64bc0b4670 2024-03-25T18:32:37.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:37.262+0000 7f64a3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:37.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:37.262+0000 7f64a3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:37.263 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:32:37.263 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:37.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:37.264+0000 7f64c1b82640 10 librados: watch_flush enter 2024-03-25T18:32:37.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:37.264+0000 7f64c1b82640 10 librados: watch_flush exit 2024-03-25T18:32:37.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:37.264+0000 7f64c1b82640 20 client.4532.objecter shutdown clearing up homeless session... 2024-03-25T18:32:37.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:37.264+0000 7f64c1b82640 10 client.4532.objecter successfully canceled tick 2024-03-25T18:32:37.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:37.264+0000 7f64c1b82640 1 -- 172.21.15.3:0/3907551109 >> v1:172.21.15.3:6800/4195140966 conn(0x7f64a806df70 legacy=0x7f64a8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:37.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:37.264+0000 7f64c1b82640 10 monclient: shutdown 2024-03-25T18:32:37.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:37.264+0000 7f64c1b82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:37.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:37.264+0000 7f64c1b82640 1 -- 172.21.15.3:0/3907551109 >> v1:172.21.15.3:6790/0 conn(0x7f64bc0b4670 legacy=0x7f64bc0b1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:37.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:37.264+0000 7f64c1b82640 1 -- 172.21.15.3:0/3907551109 shutdown_connections 2024-03-25T18:32:37.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:37.264+0000 7f64c1b82640 1 -- 172.21.15.3:0/3907551109 >> 172.21.15.3:0/3907551109 conn(0x7f64bc0617a0 msgr2=0x7f64bc140760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:37.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:37.264+0000 7f64c1b82640 1 -- 172.21.15.3:0/3907551109 shutdown_connections 2024-03-25T18:32:37.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:37.264+0000 7f64c1b82640 1 -- 172.21.15.3:0/3907551109 wait complete. 2024-03-25T18:32:37.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:37.264+0000 7f64c1b82640 1 librados: shutdown 2024-03-25T18:32:37.278 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:42.719 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:07:56.448 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:42.768 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:07:56.497 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:42.803 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:07:56.532 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:42.831 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:07:56.523 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:42.834 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:07:56.563 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:42.837 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:07:56.529 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:42.847 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:07:56.576 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:42.877 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:07:56.606 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:42.884 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:07:56.576 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:44.085 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:44.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.205+0000 7fd49d796640 10 monclient: get_monmap_and_config 2024-03-25T18:32:44.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.205+0000 7fd49d796640 10 monclient: build_initial_monmap 2024-03-25T18:32:44.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.205+0000 7fd49d796640 10 monclient: monmap: 2024-03-25T18:32:44.207 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:32:44.207 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:44.207 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:32:44.206521+0000 2024-03-25T18:32:44.207 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:32:44.206521+0000 2024-03-25T18:32:44.207 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:44.207 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:44.207 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:32:44.207 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:32:44.207 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:32:44.207 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:44.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.206+0000 7fd49d796640 1 Processor -- start 2024-03-25T18:32:44.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.206+0000 7fd49d796640 1 -- start start 2024-03-25T18:32:44.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.206+0000 7fd49d796640 10 monclient: init 2024-03-25T18:32:44.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.207+0000 7fd49d796640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:44.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.207+0000 7fd49d796640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:32:44.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.207+0000 7fd49d796640 10 monclient(hunting): picked mon.noname-c con 0x7fd4980b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:32:44.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.207+0000 7fd49d796640 10 monclient(hunting): picked mon.noname-b con 0x7fd4980b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:32:44.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.207+0000 7fd49d796640 10 monclient(hunting): picked mon.noname-a con 0x7fd49813c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:32:44.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.207+0000 7fd49d796640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4980ab0e0 con 0x7fd49813c480 2024-03-25T18:32:44.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.207+0000 7fd49d796640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4980ab3c0 con 0x7fd4980b4ff0 2024-03-25T18:32:44.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.207+0000 7fd49d796640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4980a9f30 con 0x7fd4980b4670 2024-03-25T18:32:44.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.207+0000 7fd49d796640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:44.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.207+0000 7fd49d796640 10 monclient(hunting): authenticate will time out at 7818.467536s 2024-03-25T18:32:44.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.218+0000 7fd496ffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd4980b4ff0 0x7fd49813bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:41802/0 (socket says 172.21.15.3:41802) 2024-03-25T18:32:44.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.218+0000 7fd496ffd640 1 -- 172.21.15.3:0/2495936607 learned_addr learned my addr 172.21.15.3:0/2495936607 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:32:44.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.228+0000 7fd495ffb640 1 -- 172.21.15.3:0/2495936607 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 371986681 0 0) 0x7fd4980a9f30 con 0x7fd4980b4670 2024-03-25T18:32:44.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.228+0000 7fd495ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:44.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.228+0000 7fd495ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:44.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.228+0000 7fd495ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:44.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.228+0000 7fd495ffb640 10 monclient(hunting): my global_id is 4535 2024-03-25T18:32:44.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.228+0000 7fd495ffb640 1 -- 172.21.15.3:0/2495936607 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd480003520 con 0x7fd4980b4670 2024-03-25T18:32:44.235 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.234+0000 7fd495ffb640 1 -- 172.21.15.3:0/2495936607 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 413621723 0 0) 0x7fd4980ab3c0 con 0x7fd4980b4ff0 2024-03-25T18:32:44.235 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.234+0000 7fd495ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:44.236 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.234+0000 7fd495ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:44.236 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.234+0000 7fd495ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:44.236 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.234+0000 7fd495ffb640 10 monclient(hunting): my global_id is 4531 2024-03-25T18:32:44.236 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.234+0000 7fd495ffb640 1 -- 172.21.15.3:0/2495936607 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4980a9f30 con 0x7fd4980b4ff0 2024-03-25T18:32:44.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.237+0000 7fd495ffb640 1 -- 172.21.15.3:0/2495936607 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1977630861 0 0) 0x7fd480003520 con 0x7fd4980b4670 2024-03-25T18:32:44.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.237+0000 7fd495ffb640 1 -- 172.21.15.3:0/2495936607 >> v1:172.21.15.16:6789/0 conn(0x7fd4980b4ff0 legacy=0x7fd49813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:44.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.237+0000 7fd495ffb640 1 -- 172.21.15.3:0/2495936607 >> v1:172.21.15.3:6789/0 conn(0x7fd49813c480 legacy=0x7fd49814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:44.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.237+0000 7fd495ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:32:44.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.237+0000 7fd495ffb640 1 monclient: found mon.noname-b 2024-03-25T18:32:44.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.237+0000 7fd495ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:44.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.237+0000 7fd495ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-25T18:32:44.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.237+0000 7fd495ffb640 1 -- 172.21.15.3:0/2495936607 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4980a9b70 con 0x7fd4980b4670 2024-03-25T18:32:44.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.237+0000 7fd495ffb640 10 monclient: _finish_auth 0 2024-03-25T18:32:44.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.237+0000 7fd495ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:32:44.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.237+0000 7fd495ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:44.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.237+0000 7fd495ffb640 1 -- 172.21.15.3:0/2495936607 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd48c001690 con 0x7fd4980b4670 2024-03-25T18:32:44.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.237+0000 7fd49d796640 5 monclient: authenticate success, global_id 4535 2024-03-25T18:32:44.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.237+0000 7fd49d796640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:44.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.237+0000 7fd495ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:44.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.237+0000 7fd495ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:44.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.237+0000 7fd495ffb640 10 monclient: dump: 2024-03-25T18:32:44.240 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:44.240 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:44.240 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:44.240 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:44.240 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:44.240 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:44.240 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:44.240 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:44.241 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:44.241 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:44.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.237+0000 7fd49d796640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:44.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.246+0000 7fd495ffb640 1 -- 172.21.15.3:0/2495936607 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd48c0017f0 con 0x7fd4980b4670 2024-03-25T18:32:44.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.246+0000 7fd495ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:44.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.246+0000 7fd495ffb640 1 -- 172.21.15.3:0/2495936607 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd48c001c00 con 0x7fd4980b4670 2024-03-25T18:32:44.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.246+0000 7fd495ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:44.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.246+0000 7fd495ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:44.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.246+0000 7fd495ffb640 10 monclient: dump: 2024-03-25T18:32:44.247 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:44.247 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:44.247 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:44.247 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:44.247 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:44.248 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:44.248 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:44.248 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:44.248 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:44.248 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:44.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.246+0000 7fd49d796640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:44.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.246+0000 7fd49d796640 4 set_mon_vals no callback set 2024-03-25T18:32:44.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.246+0000 7fd49d796640 10 monclient: shutdown 2024-03-25T18:32:44.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.247+0000 7fd49d796640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:44.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.247+0000 7fd49d796640 1 -- 172.21.15.3:0/2495936607 >> v1:172.21.15.3:6790/0 conn(0x7fd4980b4670 legacy=0x7fd4981425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:44.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.247+0000 7fd49d796640 1 -- 172.21.15.3:0/2495936607 shutdown_connections 2024-03-25T18:32:44.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.247+0000 7fd49d796640 1 -- 172.21.15.3:0/2495936607 >> 172.21.15.3:0/2495936607 conn(0x7fd4980617a0 msgr2=0x7fd498061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:44.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.247+0000 7fd49d796640 1 -- 172.21.15.3:0/2495936607 shutdown_connections 2024-03-25T18:32:44.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.247+0000 7fd49d796640 1 -- 172.21.15.3:0/2495936607 wait complete. 2024-03-25T18:32:44.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.247+0000 7fd49d796640 10 monclient: build_initial_monmap 2024-03-25T18:32:44.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.247+0000 7fd49d796640 10 monclient: monmap: 2024-03-25T18:32:44.250 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:32:44.250 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:44.250 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:32:44.251 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:32:44.251 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:44.251 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:32:44.251 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:32:44.251 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:32:44.251 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:32:44.251 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:44.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.248+0000 7fd49d796640 1 librados: starting msgr at 2024-03-25T18:32:44.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.248+0000 7fd49d796640 1 librados: starting objecter 2024-03-25T18:32:44.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.248+0000 7fd49d796640 1 Processor -- start 2024-03-25T18:32:44.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.248+0000 7fd49d796640 1 -- start start 2024-03-25T18:32:44.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.248+0000 7fd49d796640 1 librados: setting wanted keys 2024-03-25T18:32:44.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.248+0000 7fd49d796640 1 librados: calling monclient init 2024-03-25T18:32:44.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.248+0000 7fd49d796640 10 monclient: init 2024-03-25T18:32:44.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.248+0000 7fd49d796640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:44.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.248+0000 7fd49d796640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:32:44.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.248+0000 7fd49d796640 10 monclient(hunting): picked mon.noname-a con 0x7fd4980b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:32:44.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.249+0000 7fd49d796640 10 monclient(hunting): picked mon.noname-c con 0x7fd4980b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:32:44.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.249+0000 7fd49d796640 10 monclient(hunting): picked mon.noname-b con 0x7fd49813c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:32:44.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.249+0000 7fd49d796640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd49813d2f0 con 0x7fd4980b4670 2024-03-25T18:32:44.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.249+0000 7fd49d796640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd498143810 con 0x7fd49813c480 2024-03-25T18:32:44.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.249+0000 7fd49d796640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd49813e190 con 0x7fd4980b4ff0 2024-03-25T18:32:44.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.249+0000 7fd49d796640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:44.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.249+0000 7fd49d796640 10 monclient(hunting): authenticate will time out at 7818.509413s 2024-03-25T18:32:44.271 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.270+0000 7fd496ffd640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd4980b4670 0x7fd49813cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:37838/0 (socket says 172.21.15.3:37838) 2024-03-25T18:32:44.271 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.270+0000 7fd496ffd640 1 -- 172.21.15.3:0/3863225333 learned_addr learned my addr 172.21.15.3:0/3863225333 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:32:44.271 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.270+0000 7fd47f7fe640 10 client.?.objecter ms_handle_connect 0x7fd4980b4670 2024-03-25T18:32:44.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.270+0000 7fd47f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:44.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.277+0000 7fd47f7fe640 10 client.?.objecter ms_handle_connect 0x7fd49813c480 2024-03-25T18:32:44.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.277+0000 7fd47f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:44.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.279+0000 7fd47f7fe640 10 client.?.objecter ms_handle_connect 0x7fd4980b4ff0 2024-03-25T18:32:44.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.279+0000 7fd47f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:44.281 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.280+0000 7fd47f7fe640 1 -- 172.21.15.3:0/3863225333 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079104684 0 0) 0x7fd49813d2f0 con 0x7fd4980b4670 2024-03-25T18:32:44.281 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.280+0000 7fd47f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:44.281 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.280+0000 7fd47f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:44.281 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.280+0000 7fd47f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:44.282 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.280+0000 7fd47f7fe640 10 monclient(hunting): my global_id is 4581 2024-03-25T18:32:44.282 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.280+0000 7fd47f7fe640 1 -- 172.21.15.3:0/3863225333 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd484004eb0 con 0x7fd4980b4670 2024-03-25T18:32:44.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.294+0000 7fd47f7fe640 1 -- 172.21.15.3:0/3863225333 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2989146126 0 0) 0x7fd498143810 con 0x7fd49813c480 2024-03-25T18:32:44.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.294+0000 7fd47f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:44.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.294+0000 7fd47f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:44.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.294+0000 7fd47f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:44.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.294+0000 7fd47f7fe640 10 monclient(hunting): my global_id is 4534 2024-03-25T18:32:44.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.294+0000 7fd47f7fe640 1 -- 172.21.15.3:0/3863225333 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd49813d2f0 con 0x7fd49813c480 2024-03-25T18:32:44.296 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.294+0000 7fd47f7fe640 1 -- 172.21.15.3:0/3863225333 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159100543 0 0) 0x7fd49813e190 con 0x7fd4980b4ff0 2024-03-25T18:32:44.296 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.294+0000 7fd47f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:44.296 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.294+0000 7fd47f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:44.296 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.294+0000 7fd47f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:44.296 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.294+0000 7fd47f7fe640 10 monclient(hunting): my global_id is 4538 2024-03-25T18:32:44.296 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.294+0000 7fd47f7fe640 1 -- 172.21.15.3:0/3863225333 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd498143810 con 0x7fd4980b4ff0 2024-03-25T18:32:44.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.300+0000 7fd47f7fe640 1 -- 172.21.15.3:0/3863225333 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 884732326 0 0) 0x7fd484004eb0 con 0x7fd4980b4670 2024-03-25T18:32:44.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.300+0000 7fd47f7fe640 1 -- 172.21.15.3:0/3863225333 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd49813e190 con 0x7fd4980b4670 2024-03-25T18:32:44.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.300+0000 7fd47f7fe640 1 -- 172.21.15.3:0/3863225333 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd4880019b0 con 0x7fd4980b4670 2024-03-25T18:32:44.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.300+0000 7fd47f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:44.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.300+0000 7fd47f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:44.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.300+0000 7fd47f7fe640 10 monclient(hunting): dump: 2024-03-25T18:32:44.301 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:44.302 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:44.302 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:44.302 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:44.302 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:44.302 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:44.302 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:44.302 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:44.302 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:44.302 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:44.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.300+0000 7fd47f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:44.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.306+0000 7fd47f7fe640 1 -- 172.21.15.3:0/3863225333 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1571612700 0 0) 0x7fd498143810 con 0x7fd4980b4ff0 2024-03-25T18:32:44.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.306+0000 7fd47f7fe640 1 -- 172.21.15.3:0/3863225333 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd484004eb0 con 0x7fd4980b4ff0 2024-03-25T18:32:44.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.306+0000 7fd47f7fe640 1 -- 172.21.15.3:0/3863225333 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd48c001b00 con 0x7fd4980b4ff0 2024-03-25T18:32:44.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.306+0000 7fd47f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:44.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.306+0000 7fd47f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:44.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.306+0000 7fd47f7fe640 10 monclient(hunting): dump: 2024-03-25T18:32:44.308 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:44.308 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:44.308 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:44.308 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:44.308 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:44.308 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:44.308 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:44.308 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:44.308 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:44.308 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:44.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.313+0000 7fd47f7fe640 1 -- 172.21.15.3:0/3863225333 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 228904202 0 0) 0x7fd49813e190 con 0x7fd4980b4670 2024-03-25T18:32:44.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.313+0000 7fd47f7fe640 1 -- 172.21.15.3:0/3863225333 >> v1:172.21.15.3:6790/0 conn(0x7fd4980b4ff0 legacy=0x7fd49813ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:44.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.313+0000 7fd47f7fe640 1 -- 172.21.15.3:0/3863225333 >> v1:172.21.15.16:6789/0 conn(0x7fd49813c480 legacy=0x7fd4981e2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:44.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.313+0000 7fd47f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:32:44.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.313+0000 7fd47f7fe640 1 monclient: found mon.a 2024-03-25T18:32:44.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.313+0000 7fd47f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:44.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.313+0000 7fd47f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:32:44.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.313+0000 7fd47f7fe640 1 -- 172.21.15.3:0/3863225333 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd49813e470 con 0x7fd4980b4670 2024-03-25T18:32:44.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.313+0000 7fd47f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:32:44.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.313+0000 7fd47f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:32:44.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.313+0000 7fd47f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:44.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.313+0000 7fd49d796640 5 monclient: authenticate success, global_id 4581 2024-03-25T18:32:44.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.313+0000 7fd49d796640 10 monclient: _renew_subs 2024-03-25T18:32:44.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.313+0000 7fd49d796640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:32:44.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.313+0000 7fd49d796640 1 -- 172.21.15.3:0/3863225333 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd498145e80 con 0x7fd4980b4670 2024-03-25T18:32:44.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.313+0000 7fd49d796640 10 client.4581.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:44.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.313+0000 7fd49d796640 10 monclient: _renew_subs 2024-03-25T18:32:44.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.313+0000 7fd49d796640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:32:44.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.313+0000 7fd49d796640 1 -- 172.21.15.3:0/3863225333 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd49813f800 con 0x7fd4980b4670 2024-03-25T18:32:44.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.313+0000 7fd49d796640 1 librados: init done 2024-03-25T18:32:44.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.314+0000 7fd49d796640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:44.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.314+0000 7fd49d796640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:44.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.314+0000 7fd49d796640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:32:44.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.314+0000 7fd49d796640 1 -- 172.21.15.3:0/3863225333 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd49813e890 con 0x7fd4980b4670 2024-03-25T18:32:44.649 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.647+0000 7fd47f7fe640 1 -- 172.21.15.3:0/3863225333 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4880047e0 con 0x7fd4980b4670 2024-03-25T18:32:44.649 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.647+0000 7fd47f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:44.649 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.647+0000 7fd47f7fe640 1 -- 172.21.15.3:0/3863225333 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd488004990 con 0x7fd4980b4670 2024-03-25T18:32:44.649 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.647+0000 7fd47f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:44.649 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.647+0000 7fd47f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:44.649 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.647+0000 7fd47f7fe640 10 monclient: dump: 2024-03-25T18:32:44.649 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:44.649 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:44.649 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:44.649 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:44.649 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:44.650 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:44.650 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:44.650 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:44.650 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:44.650 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:44.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.647+0000 7fd4957fa640 4 set_mon_vals no callback set 2024-03-25T18:32:44.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.648+0000 7fd47f7fe640 1 -- 172.21.15.3:0/3863225333 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7fd488004bb0 con 0x7fd4980b4670 2024-03-25T18:32:44.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.649+0000 7fd4967fc640 10 monclient: get_auth_request con 0x7fd48406df70 auth_method 0 2024-03-25T18:32:44.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.649+0000 7fd47f7fe640 10 client.4581.objecter ms_handle_connect 0x7fd48406df70 2024-03-25T18:32:44.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.654+0000 7fd47f7fe640 1 -- 172.21.15.3:0/3863225333 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7fd488054700 con 0x7fd4980b4670 2024-03-25T18:32:44.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.654+0000 7fd47f7fe640 10 client.4581.objecter ms_dispatch 0x7fd498000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:44.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.654+0000 7fd47f7fe640 3 client.4581.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:44.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.654+0000 7fd47f7fe640 3 client.4581.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:44.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.654+0000 7fd47f7fe640 20 client.4581.objecter dump_active .. 0 homeless 2024-03-25T18:32:44.656 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.654+0000 7fd47f7fe640 1 -- 172.21.15.3:0/3863225333 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd488050210 con 0x7fd4980b4670 2024-03-25T18:32:44.656 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.654+0000 7fd47f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:44.656 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.654+0000 7fd47f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:44.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.890+0000 7fd49d796640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:44.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.890+0000 7fd49d796640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:44.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.890+0000 7fd49d796640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:32:44.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.890+0000 7fd49d796640 1 -- 172.21.15.3:0/3863225333 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd49806dbd0 con 0x7fd4980b4670 2024-03-25T18:32:44.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.932+0000 7fd47f7fe640 1 -- 172.21.15.3:0/3863225333 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7fd488052140 con 0x7fd4980b4670 2024-03-25T18:32:44.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.933+0000 7fd47f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:44.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.933+0000 7fd47f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:44.934 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:32:44.934 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:44.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.935+0000 7fd49d796640 10 librados: watch_flush enter 2024-03-25T18:32:44.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.935+0000 7fd49d796640 10 librados: watch_flush exit 2024-03-25T18:32:44.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.935+0000 7fd49d796640 20 client.4581.objecter shutdown clearing up homeless session... 2024-03-25T18:32:44.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.935+0000 7fd49d796640 10 client.4581.objecter successfully canceled tick 2024-03-25T18:32:44.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.935+0000 7fd49d796640 1 -- 172.21.15.3:0/3863225333 >> v1:172.21.15.3:6800/4195140966 conn(0x7fd48406df70 legacy=0x7fd484070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:44.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.935+0000 7fd49d796640 10 monclient: shutdown 2024-03-25T18:32:44.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.935+0000 7fd49d796640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:44.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.935+0000 7fd49d796640 1 -- 172.21.15.3:0/3863225333 >> v1:172.21.15.3:6789/0 conn(0x7fd4980b4670 legacy=0x7fd49813cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:44.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.936+0000 7fd49d796640 1 -- 172.21.15.3:0/3863225333 shutdown_connections 2024-03-25T18:32:44.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.936+0000 7fd49d796640 1 -- 172.21.15.3:0/3863225333 >> 172.21.15.3:0/3863225333 conn(0x7fd4980617a0 msgr2=0x7fd4981407a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:44.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.936+0000 7fd49d796640 1 -- 172.21.15.3:0/3863225333 shutdown_connections 2024-03-25T18:32:44.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.936+0000 7fd49d796640 1 -- 172.21.15.3:0/3863225333 wait complete. 2024-03-25T18:32:44.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:44.936+0000 7fd49d796640 1 librados: shutdown 2024-03-25T18:32:44.950 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:51.756 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:51.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.864+0000 7fc9f0a1e640 10 monclient: get_monmap_and_config 2024-03-25T18:32:51.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.864+0000 7fc9f0a1e640 10 monclient: build_initial_monmap 2024-03-25T18:32:51.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.864+0000 7fc9f0a1e640 10 monclient: monmap: 2024-03-25T18:32:51.865 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:32:51.866 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:51.866 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:32:51.865488+0000 2024-03-25T18:32:51.866 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:32:51.865488+0000 2024-03-25T18:32:51.866 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:51.866 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:51.866 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:32:51.866 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:32:51.866 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:32:51.866 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:51.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.865+0000 7fc9f0a1e640 1 Processor -- start 2024-03-25T18:32:51.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.865+0000 7fc9f0a1e640 1 -- start start 2024-03-25T18:32:51.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.865+0000 7fc9f0a1e640 10 monclient: init 2024-03-25T18:32:51.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.865+0000 7fc9f0a1e640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:51.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.865+0000 7fc9f0a1e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:32:51.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.865+0000 7fc9f0a1e640 10 monclient(hunting): picked mon.noname-a con 0x7fc9ec0b4e70 addr v1:172.21.15.3:6789/0 2024-03-25T18:32:51.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.865+0000 7fc9f0a1e640 10 monclient(hunting): picked mon.noname-c con 0x7fc9ec0b44b0 addr v1:172.21.15.16:6789/0 2024-03-25T18:32:51.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.865+0000 7fc9f0a1e640 10 monclient(hunting): picked mon.noname-b con 0x7fc9ec0b1b00 addr v1:172.21.15.3:6790/0 2024-03-25T18:32:51.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.865+0000 7fc9f0a1e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9ec0a96f0 con 0x7fc9ec0b4e70 2024-03-25T18:32:51.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.865+0000 7fc9f0a1e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9ec0ab0e0 con 0x7fc9ec0b44b0 2024-03-25T18:32:51.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.865+0000 7fc9f0a1e640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9ec145f70 con 0x7fc9ec0b1b00 2024-03-25T18:32:51.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.865+0000 7fc9f0a1e640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:51.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.865+0000 7fc9f0a1e640 10 monclient(hunting): authenticate will time out at 7826.126267s 2024-03-25T18:32:51.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.875+0000 7fc9e3fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc9ec0b44b0 0x7fc9ec145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:53258/0 (socket says 172.21.15.3:53258) 2024-03-25T18:32:51.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.875+0000 7fc9e3fff640 1 -- 172.21.15.3:0/1987176125 learned_addr learned my addr 172.21.15.3:0/1987176125 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:32:51.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.886+0000 7fc9e9d9b640 1 -- 172.21.15.3:0/1987176125 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2494968825 0 0) 0x7fc9ec0ab0e0 con 0x7fc9ec0b44b0 2024-03-25T18:32:51.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.886+0000 7fc9e9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:32:51.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.886+0000 7fc9e9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:51.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.886+0000 7fc9e9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:51.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.886+0000 7fc9e9d9b640 10 monclient(hunting): my global_id is 4537 2024-03-25T18:32:51.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.886+0000 7fc9e9d9b640 1 -- 172.21.15.3:0/1987176125 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9d8003520 con 0x7fc9ec0b44b0 2024-03-25T18:32:51.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.892+0000 7fc9e9d9b640 1 -- 172.21.15.3:0/1987176125 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 523281346 0 0) 0x7fc9d8003520 con 0x7fc9ec0b44b0 2024-03-25T18:32:51.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.892+0000 7fc9e9d9b640 1 -- 172.21.15.3:0/1987176125 >> v1:172.21.15.3:6790/0 conn(0x7fc9ec0b1b00 legacy=0x7fc9ec14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:51.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.892+0000 7fc9e9d9b640 1 -- 172.21.15.3:0/1987176125 >> v1:172.21.15.3:6789/0 conn(0x7fc9ec0b4e70 legacy=0x7fc9ec13f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:51.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.892+0000 7fc9e9d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:32:51.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.892+0000 7fc9e9d9b640 1 monclient: found mon.noname-c 2024-03-25T18:32:51.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.892+0000 7fc9e9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:51.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.892+0000 7fc9e9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:32:51.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.892+0000 7fc9e9d9b640 1 -- 172.21.15.3:0/1987176125 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9ec0a9b80 con 0x7fc9ec0b44b0 2024-03-25T18:32:51.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.892+0000 7fc9e9d9b640 10 monclient: _finish_auth 0 2024-03-25T18:32:51.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.892+0000 7fc9e9d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:32:51.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.892+0000 7fc9e9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:51.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.892+0000 7fc9e9d9b640 1 -- 172.21.15.3:0/1987176125 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc9d4001690 con 0x7fc9ec0b44b0 2024-03-25T18:32:51.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.892+0000 7fc9e9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:51.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.892+0000 7fc9e9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:32:51.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.892+0000 7fc9e9d9b640 10 monclient: dump: 2024-03-25T18:32:51.895 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:51.895 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:51.895 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:51.895 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:51.895 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:51.895 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:51.895 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:51.895 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:51.895 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:51.895 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:51.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.892+0000 7fc9f0a1e640 5 monclient: authenticate success, global_id 4537 2024-03-25T18:32:51.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.892+0000 7fc9f0a1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:51.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.906+0000 7fc9e9d9b640 1 -- 172.21.15.3:0/1987176125 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9d40017f0 con 0x7fc9ec0b44b0 2024-03-25T18:32:51.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.906+0000 7fc9e9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:51.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.906+0000 7fc9e9d9b640 1 -- 172.21.15.3:0/1987176125 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc9d4001c40 con 0x7fc9ec0b44b0 2024-03-25T18:32:51.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.906+0000 7fc9e9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:51.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.906+0000 7fc9e9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:51.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.906+0000 7fc9e9d9b640 10 monclient: dump: 2024-03-25T18:32:51.907 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:51.908 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:51.908 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:51.908 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:51.908 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:51.908 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:51.908 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:51.908 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:51.908 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:51.908 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:51.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.906+0000 7fc9f0a1e640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:51.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.906+0000 7fc9f0a1e640 4 set_mon_vals no callback set 2024-03-25T18:32:51.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.907+0000 7fc9f0a1e640 10 monclient: shutdown 2024-03-25T18:32:51.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.907+0000 7fc9f0a1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:51.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.907+0000 7fc9f0a1e640 1 -- 172.21.15.3:0/1987176125 >> v1:172.21.15.16:6789/0 conn(0x7fc9ec0b44b0 legacy=0x7fc9ec145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:51.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.907+0000 7fc9f0a1e640 1 -- 172.21.15.3:0/1987176125 shutdown_connections 2024-03-25T18:32:51.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.907+0000 7fc9f0a1e640 1 -- 172.21.15.3:0/1987176125 >> 172.21.15.3:0/1987176125 conn(0x7fc9ec0617a0 msgr2=0x7fc9ec061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:51.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.907+0000 7fc9f0a1e640 1 -- 172.21.15.3:0/1987176125 shutdown_connections 2024-03-25T18:32:51.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.907+0000 7fc9f0a1e640 1 -- 172.21.15.3:0/1987176125 wait complete. 2024-03-25T18:32:51.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.908+0000 7fc9f0a1e640 10 monclient: build_initial_monmap 2024-03-25T18:32:51.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.908+0000 7fc9f0a1e640 10 monclient: monmap: 2024-03-25T18:32:51.911 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:32:51.911 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:51.911 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:32:51.911 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:32:51.911 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:51.911 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:32:51.911 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:32:51.912 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:32:51.912 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:32:51.912 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:51.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.908+0000 7fc9f0a1e640 1 librados: starting msgr at 2024-03-25T18:32:51.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.908+0000 7fc9f0a1e640 1 librados: starting objecter 2024-03-25T18:32:51.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.908+0000 7fc9f0a1e640 1 Processor -- start 2024-03-25T18:32:51.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.908+0000 7fc9f0a1e640 1 -- start start 2024-03-25T18:32:51.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.908+0000 7fc9f0a1e640 1 librados: setting wanted keys 2024-03-25T18:32:51.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.908+0000 7fc9f0a1e640 1 librados: calling monclient init 2024-03-25T18:32:51.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.908+0000 7fc9f0a1e640 10 monclient: init 2024-03-25T18:32:51.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.909+0000 7fc9f0a1e640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:51.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.909+0000 7fc9f0a1e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:32:51.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.909+0000 7fc9f0a1e640 10 monclient(hunting): picked mon.noname-b con 0x7fc9ec0b1b00 addr v1:172.21.15.16:6789/0 2024-03-25T18:32:51.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.909+0000 7fc9f0a1e640 10 monclient(hunting): picked mon.noname-a con 0x7fc9ec0b44b0 addr v1:172.21.15.3:6789/0 2024-03-25T18:32:51.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.909+0000 7fc9f0a1e640 10 monclient(hunting): picked mon.noname-c con 0x7fc9ec0b4e70 addr v1:172.21.15.3:6790/0 2024-03-25T18:32:51.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.909+0000 7fc9f0a1e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9ec131c10 con 0x7fc9ec0b44b0 2024-03-25T18:32:51.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.909+0000 7fc9f0a1e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9ec1dabe0 con 0x7fc9ec0b1b00 2024-03-25T18:32:51.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.909+0000 7fc9f0a1e640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9ec1d76e0 con 0x7fc9ec0b4e70 2024-03-25T18:32:51.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.909+0000 7fc9f0a1e640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:51.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.909+0000 7fc9f0a1e640 10 monclient(hunting): authenticate will time out at 7826.169700s 2024-03-25T18:32:51.924 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.923+0000 7fc9e3fff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fc9ec0b44b0 0x7fc9ec1db100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:38476/0 (socket says 172.21.15.3:38476) 2024-03-25T18:32:51.924 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.923+0000 7fc9e3fff640 1 -- 172.21.15.3:0/1980166017 learned_addr learned my addr 172.21.15.3:0/1980166017 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:32:51.925 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.924+0000 7fc9e2ffd640 10 client.?.objecter ms_handle_connect 0x7fc9ec0b4e70 2024-03-25T18:32:51.925 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.924+0000 7fc9e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:51.925 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.925+0000 7fc9e2ffd640 10 client.?.objecter ms_handle_connect 0x7fc9ec0b44b0 2024-03-25T18:32:51.926 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.925+0000 7fc9e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:51.929 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.928+0000 7fc9e2ffd640 10 client.?.objecter ms_handle_connect 0x7fc9ec0b1b00 2024-03-25T18:32:51.929 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.928+0000 7fc9e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:51.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.935+0000 7fc9e2ffd640 1 -- 172.21.15.3:0/1980166017 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448015440 0 0) 0x7fc9ec131c10 con 0x7fc9ec0b44b0 2024-03-25T18:32:51.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.935+0000 7fc9e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:51.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.935+0000 7fc9e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:51.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.935+0000 7fc9e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:51.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.935+0000 7fc9e2ffd640 10 monclient(hunting): my global_id is 4587 2024-03-25T18:32:51.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.935+0000 7fc9e2ffd640 1 -- 172.21.15.3:0/1980166017 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9cc004eb0 con 0x7fc9ec0b44b0 2024-03-25T18:32:51.946 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.945+0000 7fc9e2ffd640 1 -- 172.21.15.3:0/1980166017 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990482578 0 0) 0x7fc9ec1dabe0 con 0x7fc9ec0b1b00 2024-03-25T18:32:51.946 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.945+0000 7fc9e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:51.946 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.945+0000 7fc9e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:51.946 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.945+0000 7fc9e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:51.946 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.945+0000 7fc9e2ffd640 10 monclient(hunting): my global_id is 4540 2024-03-25T18:32:51.946 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.945+0000 7fc9e2ffd640 1 -- 172.21.15.3:0/1980166017 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9ec131c10 con 0x7fc9ec0b1b00 2024-03-25T18:32:51.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.952+0000 7fc9e2ffd640 1 -- 172.21.15.3:0/1980166017 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2190863993 0 0) 0x7fc9ec1d76e0 con 0x7fc9ec0b4e70 2024-03-25T18:32:51.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.952+0000 7fc9e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:51.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.952+0000 7fc9e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:51.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.952+0000 7fc9e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:51.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.952+0000 7fc9e2ffd640 10 monclient(hunting): my global_id is 4544 2024-03-25T18:32:51.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.952+0000 7fc9e2ffd640 1 -- 172.21.15.3:0/1980166017 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9ec1dabe0 con 0x7fc9ec0b4e70 2024-03-25T18:32:51.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.955+0000 7fc9e2ffd640 1 -- 172.21.15.3:0/1980166017 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4159291723 0 0) 0x7fc9ec131c10 con 0x7fc9ec0b1b00 2024-03-25T18:32:51.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.956+0000 7fc9e2ffd640 1 -- 172.21.15.3:0/1980166017 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9ec1d76e0 con 0x7fc9ec0b1b00 2024-03-25T18:32:51.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.956+0000 7fc9e2ffd640 1 -- 172.21.15.3:0/1980166017 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc9dc001810 con 0x7fc9ec0b1b00 2024-03-25T18:32:51.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.956+0000 7fc9e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:51.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.956+0000 7fc9e2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:51.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.956+0000 7fc9e2ffd640 10 monclient(hunting): dump: 2024-03-25T18:32:51.958 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:51.958 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:51.958 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:51.958 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:51.958 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:51.958 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:51.958 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:51.958 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:51.958 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:51.958 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:51.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.956+0000 7fc9e2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:51.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.956+0000 7fc9e2ffd640 1 -- 172.21.15.3:0/1980166017 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 210753151 0 0) 0x7fc9cc004eb0 con 0x7fc9ec0b44b0 2024-03-25T18:32:51.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.956+0000 7fc9e2ffd640 1 -- 172.21.15.3:0/1980166017 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9ec131c10 con 0x7fc9ec0b44b0 2024-03-25T18:32:51.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.956+0000 7fc9e2ffd640 1 -- 172.21.15.3:0/1980166017 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc9d4001b40 con 0x7fc9ec0b44b0 2024-03-25T18:32:51.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.956+0000 7fc9e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:51.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.956+0000 7fc9e2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:51.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.956+0000 7fc9e2ffd640 10 monclient(hunting): dump: 2024-03-25T18:32:51.959 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:51.959 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:51.959 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:51.959 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:51.959 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:51.959 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:51.959 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:51.959 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:51.959 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:51.959 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:51.968 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.966+0000 7fc9e2ffd640 1 -- 172.21.15.3:0/1980166017 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1359324729 0 0) 0x7fc9ec131c10 con 0x7fc9ec0b44b0 2024-03-25T18:32:51.968 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.966+0000 7fc9e2ffd640 1 -- 172.21.15.3:0/1980166017 >> v1:172.21.15.3:6790/0 conn(0x7fc9ec0b4e70 legacy=0x7fc9ec1d7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:51.968 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.966+0000 7fc9e2ffd640 1 -- 172.21.15.3:0/1980166017 >> v1:172.21.15.16:6789/0 conn(0x7fc9ec0b1b00 legacy=0x7fc9ec1300c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:51.968 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.966+0000 7fc9e2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:32:51.968 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.966+0000 7fc9e2ffd640 1 monclient: found mon.a 2024-03-25T18:32:51.968 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.966+0000 7fc9e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:51.968 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.966+0000 7fc9e2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:32:51.968 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.966+0000 7fc9e2ffd640 1 -- 172.21.15.3:0/1980166017 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9ec1d7870 con 0x7fc9ec0b44b0 2024-03-25T18:32:51.968 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.966+0000 7fc9e2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:32:51.968 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.966+0000 7fc9e2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:32:51.968 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.966+0000 7fc9e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:51.968 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.966+0000 7fc9f0a1e640 5 monclient: authenticate success, global_id 4587 2024-03-25T18:32:51.969 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.966+0000 7fc9f0a1e640 10 monclient: _renew_subs 2024-03-25T18:32:51.969 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.966+0000 7fc9f0a1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:32:51.969 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.966+0000 7fc9f0a1e640 1 -- 172.21.15.3:0/1980166017 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9ec12f540 con 0x7fc9ec0b44b0 2024-03-25T18:32:51.969 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.966+0000 7fc9f0a1e640 10 client.4587.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:51.969 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.966+0000 7fc9f0a1e640 10 monclient: _renew_subs 2024-03-25T18:32:51.969 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.966+0000 7fc9f0a1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:32:51.969 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.966+0000 7fc9f0a1e640 1 -- 172.21.15.3:0/1980166017 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9ec1309b0 con 0x7fc9ec0b44b0 2024-03-25T18:32:51.969 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.966+0000 7fc9f0a1e640 1 librados: init done 2024-03-25T18:32:51.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.968+0000 7fc9f0a1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:51.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.968+0000 7fc9f0a1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:51.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.968+0000 7fc9f0a1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:32:51.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:51.968+0000 7fc9f0a1e640 1 -- 172.21.15.3:0/1980166017 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9ec04f730 con 0x7fc9ec0b44b0 2024-03-25T18:32:52.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.276+0000 7fc9e2ffd640 1 -- 172.21.15.3:0/1980166017 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9d4004bf0 con 0x7fc9ec0b44b0 2024-03-25T18:32:52.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.276+0000 7fc9e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:52.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.276+0000 7fc9e2ffd640 1 -- 172.21.15.3:0/1980166017 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc9d4001d80 con 0x7fc9ec0b44b0 2024-03-25T18:32:52.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.276+0000 7fc9e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:52.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.276+0000 7fc9e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:52.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.276+0000 7fc9e2ffd640 10 monclient: dump: 2024-03-25T18:32:52.278 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:52.278 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:52.278 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:52.278 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:52.278 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:52.278 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:52.278 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:52.278 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:52.279 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:52.279 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:52.279 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.276+0000 7fc9e959a640 4 set_mon_vals no callback set 2024-03-25T18:32:52.279 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.276+0000 7fc9e2ffd640 1 -- 172.21.15.3:0/1980166017 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7fc9d401abd0 con 0x7fc9ec0b44b0 2024-03-25T18:32:52.279 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.277+0000 7fc9ea59c640 10 monclient: get_auth_request con 0x7fc9cc06df50 auth_method 0 2024-03-25T18:32:52.279 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.277+0000 7fc9e2ffd640 10 client.4587.objecter ms_handle_connect 0x7fc9cc06df50 2024-03-25T18:32:52.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.284+0000 7fc9e2ffd640 1 -- 172.21.15.3:0/1980166017 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7fc9d40547f0 con 0x7fc9ec0b44b0 2024-03-25T18:32:52.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.284+0000 7fc9e2ffd640 10 client.4587.objecter ms_dispatch 0x7fc9ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:52.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.284+0000 7fc9e2ffd640 3 client.4587.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:52.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.284+0000 7fc9e2ffd640 3 client.4587.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:52.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.284+0000 7fc9e2ffd640 20 client.4587.objecter dump_active .. 0 homeless 2024-03-25T18:32:52.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.284+0000 7fc9e2ffd640 1 -- 172.21.15.3:0/1980166017 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc9d4050130 con 0x7fc9ec0b44b0 2024-03-25T18:32:52.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.284+0000 7fc9e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:52.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.284+0000 7fc9e2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:52.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.535+0000 7fc9f0a1e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:52.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.535+0000 7fc9f0a1e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:52.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.535+0000 7fc9f0a1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:32:52.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.535+0000 7fc9f0a1e640 1 -- 172.21.15.3:0/1980166017 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9ec1dad70 con 0x7fc9ec0b44b0 2024-03-25T18:32:52.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.572+0000 7fc9e2ffd640 1 -- 172.21.15.3:0/1980166017 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7fc9d4052230 con 0x7fc9ec0b44b0 2024-03-25T18:32:52.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.572+0000 7fc9e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:52.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.572+0000 7fc9e2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:52.574 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:32:52.574 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:52.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.575+0000 7fc9f0a1e640 10 librados: watch_flush enter 2024-03-25T18:32:52.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.575+0000 7fc9f0a1e640 10 librados: watch_flush exit 2024-03-25T18:32:52.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.575+0000 7fc9f0a1e640 20 client.4587.objecter shutdown clearing up homeless session... 2024-03-25T18:32:52.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.575+0000 7fc9f0a1e640 10 client.4587.objecter successfully canceled tick 2024-03-25T18:32:52.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.575+0000 7fc9f0a1e640 1 -- 172.21.15.3:0/1980166017 >> v1:172.21.15.3:6800/4195140966 conn(0x7fc9cc06df50 legacy=0x7fc9cc070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:52.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.575+0000 7fc9f0a1e640 10 monclient: shutdown 2024-03-25T18:32:52.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.575+0000 7fc9f0a1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:52.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.575+0000 7fc9f0a1e640 1 -- 172.21.15.3:0/1980166017 >> v1:172.21.15.3:6789/0 conn(0x7fc9ec0b44b0 legacy=0x7fc9ec1db100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:52.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.575+0000 7fc9f0a1e640 1 -- 172.21.15.3:0/1980166017 shutdown_connections 2024-03-25T18:32:52.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.575+0000 7fc9f0a1e640 1 -- 172.21.15.3:0/1980166017 >> 172.21.15.3:0/1980166017 conn(0x7fc9ec0617a0 msgr2=0x7fc9ec0b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:52.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.575+0000 7fc9f0a1e640 1 -- 172.21.15.3:0/1980166017 shutdown_connections 2024-03-25T18:32:52.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.575+0000 7fc9f0a1e640 1 -- 172.21.15.3:0/1980166017 wait complete. 2024-03-25T18:32:52.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:52.575+0000 7fc9f0a1e640 1 librados: shutdown 2024-03-25T18:32:52.588 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:59.395 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:59.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.513+0000 7f4db020d640 10 monclient: get_monmap_and_config 2024-03-25T18:32:59.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.513+0000 7f4db020d640 10 monclient: build_initial_monmap 2024-03-25T18:32:59.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.513+0000 7f4db020d640 10 monclient: monmap: 2024-03-25T18:32:59.515 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:32:59.515 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:59.515 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:32:59.514926+0000 2024-03-25T18:32:59.516 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:32:59.514926+0000 2024-03-25T18:32:59.516 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:59.516 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:59.516 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:32:59.516 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:32:59.516 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:32:59.516 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:59.517 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.514+0000 7f4db020d640 1 Processor -- start 2024-03-25T18:32:59.517 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.514+0000 7f4db020d640 1 -- start start 2024-03-25T18:32:59.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.514+0000 7f4db020d640 10 monclient: init 2024-03-25T18:32:59.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.515+0000 7f4db020d640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:59.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.515+0000 7f4db020d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:32:59.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.515+0000 7f4db020d640 10 monclient(hunting): picked mon.noname-b con 0x7f4da80b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:32:59.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.515+0000 7f4db020d640 10 monclient(hunting): picked mon.noname-c con 0x7f4da80b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:32:59.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.515+0000 7f4db020d640 10 monclient(hunting): picked mon.noname-a con 0x7f4da813c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:32:59.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.515+0000 7f4db020d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4da80ab0e0 con 0x7f4da813c480 2024-03-25T18:32:59.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.515+0000 7f4db020d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4da80ab3c0 con 0x7f4da80b4670 2024-03-25T18:32:59.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.516+0000 7f4db020d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4da80a9f30 con 0x7f4da80b4ff0 2024-03-25T18:32:59.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.516+0000 7f4db020d640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:59.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.516+0000 7f4db020d640 10 monclient(hunting): authenticate will time out at 7833.776369s 2024-03-25T18:32:59.526 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.525+0000 7f4dae7aa640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f4da813c480 0x7f4da814dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53846/0 (socket says 172.21.15.3:53846) 2024-03-25T18:32:59.527 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.525+0000 7f4dae7aa640 1 -- 172.21.15.3:0/265894343 learned_addr learned my addr 172.21.15.3:0/265894343 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:32:59.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.535+0000 7f4dacfa7640 1 -- 172.21.15.3:0/265894343 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2916459917 0 0) 0x7f4da80ab0e0 con 0x7f4da813c480 2024-03-25T18:32:59.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.535+0000 7f4dacfa7640 10 monclient(hunting): _negotiate 2024-03-25T18:32:59.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.535+0000 7f4dacfa7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:59.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.535+0000 7f4dacfa7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:59.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.535+0000 7f4dacfa7640 10 monclient(hunting): my global_id is 4590 2024-03-25T18:32:59.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.535+0000 7f4dacfa7640 1 -- 172.21.15.3:0/265894343 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d94003520 con 0x7f4da813c480 2024-03-25T18:32:59.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.537+0000 7f4dacfa7640 1 -- 172.21.15.3:0/265894343 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3843541999 0 0) 0x7f4da80a9f30 con 0x7f4da80b4ff0 2024-03-25T18:32:59.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.537+0000 7f4dacfa7640 10 monclient(hunting): _negotiate 2024-03-25T18:32:59.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.537+0000 7f4dacfa7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:59.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.537+0000 7f4dacfa7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:59.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.537+0000 7f4dacfa7640 10 monclient(hunting): my global_id is 4547 2024-03-25T18:32:59.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.537+0000 7f4dacfa7640 1 -- 172.21.15.3:0/265894343 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4da80ab0e0 con 0x7f4da80b4ff0 2024-03-25T18:32:59.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.547+0000 7f4dacfa7640 1 -- 172.21.15.3:0/265894343 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2002975 0 0) 0x7f4d94003520 con 0x7f4da813c480 2024-03-25T18:32:59.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.547+0000 7f4dacfa7640 1 -- 172.21.15.3:0/265894343 >> v1:172.21.15.3:6790/0 conn(0x7f4da80b4ff0 legacy=0x7f4da813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:59.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.547+0000 7f4dacfa7640 1 -- 172.21.15.3:0/265894343 >> v1:172.21.15.16:6789/0 conn(0x7f4da80b4670 legacy=0x7f4da81425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:59.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.547+0000 7f4dacfa7640 10 monclient: _finish_hunting 0 2024-03-25T18:32:59.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.547+0000 7f4dacfa7640 1 monclient: found mon.noname-a 2024-03-25T18:32:59.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.547+0000 7f4dacfa7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:59.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.547+0000 7f4dacfa7640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:32:59.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.547+0000 7f4dacfa7640 1 -- 172.21.15.3:0/265894343 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da80a9b70 con 0x7f4da813c480 2024-03-25T18:32:59.550 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.547+0000 7f4dacfa7640 10 monclient: _finish_auth 0 2024-03-25T18:32:59.550 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.547+0000 7f4dacfa7640 10 monclient: _check_auth_tickets 2024-03-25T18:32:59.550 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.547+0000 7f4dacfa7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:59.550 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.547+0000 7f4dacfa7640 1 -- 172.21.15.3:0/265894343 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4da0001a10 con 0x7f4da813c480 2024-03-25T18:32:59.550 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.547+0000 7f4db020d640 5 monclient: authenticate success, global_id 4590 2024-03-25T18:32:59.550 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.547+0000 7f4db020d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:59.550 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.547+0000 7f4dacfa7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:59.550 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.547+0000 7f4dacfa7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:59.550 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.547+0000 7f4dacfa7640 10 monclient: dump: 2024-03-25T18:32:59.550 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:59.550 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:59.550 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:59.551 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:59.551 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:59.551 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:59.551 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:59.551 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:59.551 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:59.551 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:59.551 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.547+0000 7f4db020d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:59.568 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.566+0000 7f4dacfa7640 1 -- 172.21.15.3:0/265894343 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4da0001b70 con 0x7f4da813c480 2024-03-25T18:32:59.568 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.566+0000 7f4dacfa7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:59.568 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.567+0000 7f4db020d640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:59.568 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.567+0000 7f4db020d640 4 set_mon_vals no callback set 2024-03-25T18:32:59.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.567+0000 7f4db020d640 10 monclient: shutdown 2024-03-25T18:32:59.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.567+0000 7f4db020d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:59.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.567+0000 7f4db020d640 1 -- 172.21.15.3:0/265894343 >> v1:172.21.15.3:6789/0 conn(0x7f4da813c480 legacy=0x7f4da814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:59.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.567+0000 7f4db020d640 1 -- 172.21.15.3:0/265894343 shutdown_connections 2024-03-25T18:32:59.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.567+0000 7f4db020d640 1 -- 172.21.15.3:0/265894343 >> 172.21.15.3:0/265894343 conn(0x7f4da80617a0 msgr2=0x7f4da8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:59.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.567+0000 7f4db020d640 1 -- 172.21.15.3:0/265894343 shutdown_connections 2024-03-25T18:32:59.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.567+0000 7f4db020d640 1 -- 172.21.15.3:0/265894343 wait complete. 2024-03-25T18:32:59.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.568+0000 7f4db020d640 10 monclient: build_initial_monmap 2024-03-25T18:32:59.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.568+0000 7f4db020d640 10 monclient: monmap: 2024-03-25T18:32:59.571 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:32:59.571 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:59.571 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:32:59.571 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:32:59.571 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:59.571 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:32:59.571 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:32:59.571 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:32:59.571 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:32:59.571 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:59.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.568+0000 7f4db020d640 1 librados: starting msgr at 2024-03-25T18:32:59.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.568+0000 7f4db020d640 1 librados: starting objecter 2024-03-25T18:32:59.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.569+0000 7f4db020d640 1 Processor -- start 2024-03-25T18:32:59.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.569+0000 7f4db020d640 1 -- start start 2024-03-25T18:32:59.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.569+0000 7f4db020d640 1 librados: setting wanted keys 2024-03-25T18:32:59.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.569+0000 7f4db020d640 1 librados: calling monclient init 2024-03-25T18:32:59.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.569+0000 7f4db020d640 10 monclient: init 2024-03-25T18:32:59.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.569+0000 7f4db020d640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:59.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.569+0000 7f4db020d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:32:59.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.569+0000 7f4db020d640 10 monclient(hunting): picked mon.noname-a con 0x7f4da80b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:32:59.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.569+0000 7f4db020d640 10 monclient(hunting): picked mon.noname-b con 0x7f4da80b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:32:59.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.569+0000 7f4db020d640 10 monclient(hunting): picked mon.noname-c con 0x7f4da813c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:32:59.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.569+0000 7f4db020d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4da813d9d0 con 0x7f4da80b4670 2024-03-25T18:32:59.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.569+0000 7f4db020d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4da813dc40 con 0x7f4da80b4ff0 2024-03-25T18:32:59.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.569+0000 7f4db020d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4da813c930 con 0x7f4da813c480 2024-03-25T18:32:59.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.569+0000 7f4db020d640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:59.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.569+0000 7f4db020d640 10 monclient(hunting): authenticate will time out at 7833.830285s 2024-03-25T18:32:59.582 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.581+0000 7f4dae7aa640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f4da813c480 0x7f4da81e6790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:54286/0 (socket says 172.21.15.3:54286) 2024-03-25T18:32:59.582 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.581+0000 7f4dae7aa640 1 -- 172.21.15.3:0/4202080816 learned_addr learned my addr 172.21.15.3:0/4202080816 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:32:59.586 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.585+0000 7f4d9e7fc640 10 client.?.objecter ms_handle_connect 0x7f4da80b4670 2024-03-25T18:32:59.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.585+0000 7f4d9e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:59.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.585+0000 7f4d9e7fc640 10 client.?.objecter ms_handle_connect 0x7f4da80b4ff0 2024-03-25T18:32:59.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.585+0000 7f4d9e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:59.588 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.587+0000 7f4d9e7fc640 10 client.?.objecter ms_handle_connect 0x7f4da813c480 2024-03-25T18:32:59.588 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.587+0000 7f4d9e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:59.594 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.593+0000 7f4d9e7fc640 1 -- 172.21.15.3:0/4202080816 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1520832910 0 0) 0x7f4da813dc40 con 0x7f4da80b4ff0 2024-03-25T18:32:59.594 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.593+0000 7f4d9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:59.594 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.593+0000 7f4d9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:59.594 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.593+0000 7f4d9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:59.594 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.593+0000 7f4d9e7fc640 10 monclient(hunting): my global_id is 4546 2024-03-25T18:32:59.595 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.593+0000 7f4d9e7fc640 1 -- 172.21.15.3:0/4202080816 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d8c004eb0 con 0x7f4da80b4ff0 2024-03-25T18:32:59.595 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.593+0000 7f4d9e7fc640 1 -- 172.21.15.3:0/4202080816 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1348268579 0 0) 0x7f4da813c930 con 0x7f4da813c480 2024-03-25T18:32:59.595 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.593+0000 7f4d9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:59.595 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.593+0000 7f4d9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:59.595 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.593+0000 7f4d9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:59.595 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.593+0000 7f4d9e7fc640 10 monclient(hunting): my global_id is 4550 2024-03-25T18:32:59.595 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.593+0000 7f4d9e7fc640 1 -- 172.21.15.3:0/4202080816 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4da813dc40 con 0x7f4da813c480 2024-03-25T18:32:59.603 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.601+0000 7f4d9e7fc640 1 -- 172.21.15.3:0/4202080816 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1054522469 0 0) 0x7f4da813d9d0 con 0x7f4da80b4670 2024-03-25T18:32:59.603 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.601+0000 7f4d9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:59.603 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.601+0000 7f4d9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:59.603 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.601+0000 7f4d9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:59.604 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.601+0000 7f4d9e7fc640 10 monclient(hunting): my global_id is 4593 2024-03-25T18:32:59.604 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.601+0000 7f4d9e7fc640 1 -- 172.21.15.3:0/4202080816 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4da813c930 con 0x7f4da80b4670 2024-03-25T18:32:59.604 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.602+0000 7f4d9e7fc640 1 -- 172.21.15.3:0/4202080816 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1597010889 0 0) 0x7f4d8c004eb0 con 0x7f4da80b4ff0 2024-03-25T18:32:59.604 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.602+0000 7f4d9e7fc640 1 -- 172.21.15.3:0/4202080816 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4da813d9d0 con 0x7f4da80b4ff0 2024-03-25T18:32:59.604 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.602+0000 7f4d9e7fc640 1 -- 172.21.15.3:0/4202080816 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4d98001930 con 0x7f4da80b4ff0 2024-03-25T18:32:59.604 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.602+0000 7f4d9e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:59.604 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.602+0000 7f4d9e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:59.604 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.602+0000 7f4d9e7fc640 10 monclient(hunting): dump: 2024-03-25T18:32:59.604 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:59.604 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:59.604 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:59.604 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:59.604 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:59.605 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:59.605 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:59.605 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:59.605 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:59.605 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:59.605 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.602+0000 7f4d9e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:59.605 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.604+0000 7f4d9e7fc640 1 -- 172.21.15.3:0/4202080816 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3247260106 0 0) 0x7f4da813dc40 con 0x7f4da813c480 2024-03-25T18:32:59.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.604+0000 7f4d9e7fc640 1 -- 172.21.15.3:0/4202080816 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d8c004eb0 con 0x7f4da813c480 2024-03-25T18:32:59.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.604+0000 7f4d9e7fc640 1 -- 172.21.15.3:0/4202080816 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4da0001ec0 con 0x7f4da813c480 2024-03-25T18:32:59.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.604+0000 7f4d9e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:59.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.604+0000 7f4d9e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:59.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.604+0000 7f4d9e7fc640 10 monclient(hunting): dump: 2024-03-25T18:32:59.606 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:59.606 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:59.606 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:59.606 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:59.606 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:59.607 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:59.607 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:59.607 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:59.607 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:59.607 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:59.609 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.607+0000 7f4d9e7fc640 1 -- 172.21.15.3:0/4202080816 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3473844016 0 0) 0x7f4da813d9d0 con 0x7f4da80b4ff0 2024-03-25T18:32:59.609 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.607+0000 7f4d9e7fc640 1 -- 172.21.15.3:0/4202080816 >> v1:172.21.15.3:6790/0 conn(0x7f4da813c480 legacy=0x7f4da81e6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:59.609 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.607+0000 7f4d9e7fc640 1 -- 172.21.15.3:0/4202080816 >> v1:172.21.15.3:6789/0 conn(0x7f4da80b4670 legacy=0x7f4da8133260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:59.609 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.607+0000 7f4d9e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:32:59.609 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.607+0000 7f4d9e7fc640 1 monclient: found mon.b 2024-03-25T18:32:59.609 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.607+0000 7f4d9e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:59.609 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.607+0000 7f4d9e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:32:59.609 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.607+0000 7f4d9e7fc640 1 -- 172.21.15.3:0/4202080816 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da813cc10 con 0x7f4da80b4ff0 2024-03-25T18:32:59.609 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.607+0000 7f4d9e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:32:59.609 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.607+0000 7f4d9e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:32:59.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.607+0000 7f4d9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:59.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.607+0000 7f4db020d640 5 monclient: authenticate success, global_id 4546 2024-03-25T18:32:59.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.607+0000 7f4db020d640 10 monclient: _renew_subs 2024-03-25T18:32:59.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.607+0000 7f4db020d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:32:59.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.607+0000 7f4db020d640 1 -- 172.21.15.3:0/4202080816 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4da8130c40 con 0x7f4da80b4ff0 2024-03-25T18:32:59.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.607+0000 7f4db020d640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:59.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.607+0000 7f4db020d640 10 monclient: _renew_subs 2024-03-25T18:32:59.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.607+0000 7f4db020d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:32:59.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.607+0000 7f4db020d640 1 -- 172.21.15.3:0/4202080816 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4da81461f0 con 0x7f4da80b4ff0 2024-03-25T18:32:59.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.607+0000 7f4db020d640 1 librados: init done 2024-03-25T18:32:59.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.608+0000 7f4db020d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:59.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.608+0000 7f4db020d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:59.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.608+0000 7f4db020d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:32:59.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.608+0000 7f4db020d640 1 -- 172.21.15.3:0/4202080816 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4da813e160 con 0x7f4da80b4ff0 2024-03-25T18:32:59.899 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.898+0000 7f4d9e7fc640 1 -- 172.21.15.3:0/4202080816 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d980035f0 con 0x7f4da80b4ff0 2024-03-25T18:32:59.899 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.898+0000 7f4d9e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:59.899 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.898+0000 7f4d9e7fc640 1 -- 172.21.15.3:0/4202080816 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4d980037e0 con 0x7f4da80b4ff0 2024-03-25T18:32:59.899 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.898+0000 7f4d9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:59.899 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.898+0000 7f4d9e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:59.899 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.898+0000 7f4d9e7fc640 10 monclient: dump: 2024-03-25T18:32:59.900 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:32:59.900 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:32:59.900 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:59.900 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:32:59.900 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:59.900 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:32:59.900 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:32:59.900 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:32:59.900 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:32:59.900 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:32:59.900 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.898+0000 7f4d9ffff640 4 set_mon_vals no callback set 2024-03-25T18:32:59.901 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.899+0000 7f4d9e7fc640 1 -- 172.21.15.3:0/4202080816 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f4d980039c0 con 0x7f4da80b4ff0 2024-03-25T18:32:59.901 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.899+0000 7f4dadfa9640 10 monclient: get_auth_request con 0x7f4d8c06df20 auth_method 0 2024-03-25T18:32:59.901 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.900+0000 7f4d9e7fc640 10 client.4546.objecter ms_handle_connect 0x7f4d8c06df20 2024-03-25T18:32:59.905 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.904+0000 7f4d9e7fc640 1 -- 172.21.15.3:0/4202080816 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f4d98053500 con 0x7f4da80b4ff0 2024-03-25T18:32:59.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.904+0000 7f4d9e7fc640 10 client.4546.objecter ms_dispatch 0x7f4da8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:59.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.904+0000 7f4d9e7fc640 3 client.4546.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:59.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.904+0000 7f4d9e7fc640 3 client.4546.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:59.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.904+0000 7f4d9e7fc640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-25T18:32:59.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.906+0000 7f4d9e7fc640 1 -- 172.21.15.3:0/4202080816 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4d9804f010 con 0x7f4da80b4ff0 2024-03-25T18:32:59.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.906+0000 7f4d9e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:59.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:32:59.906+0000 7f4d9e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:00.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:00.139+0000 7f4db020d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:00.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:00.139+0000 7f4db020d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:00.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:00.139+0000 7f4db020d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:33:00.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:00.139+0000 7f4db020d640 1 -- 172.21.15.3:0/4202080816 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4da813cda0 con 0x7f4da80b4ff0 2024-03-25T18:33:00.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:00.176+0000 7f4d9e7fc640 1 -- 172.21.15.3:0/4202080816 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f4d98050f40 con 0x7f4da80b4ff0 2024-03-25T18:33:00.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:00.176+0000 7f4d9e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:00.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:00.176+0000 7f4d9e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:00.177 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:33:00.177 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:00.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:00.178+0000 7f4db020d640 10 librados: watch_flush enter 2024-03-25T18:33:00.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:00.178+0000 7f4db020d640 10 librados: watch_flush exit 2024-03-25T18:33:00.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:00.178+0000 7f4db020d640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-25T18:33:00.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:00.178+0000 7f4db020d640 10 client.4546.objecter successfully canceled tick 2024-03-25T18:33:00.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:00.178+0000 7f4db020d640 1 -- 172.21.15.3:0/4202080816 >> v1:172.21.15.3:6800/4195140966 conn(0x7f4d8c06df20 legacy=0x7f4d8c0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:00.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:00.178+0000 7f4db020d640 10 monclient: shutdown 2024-03-25T18:33:00.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:00.178+0000 7f4db020d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:00.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:00.178+0000 7f4db020d640 1 -- 172.21.15.3:0/4202080816 >> v1:172.21.15.16:6789/0 conn(0x7f4da80b4ff0 legacy=0x7f4da8133970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:00.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:00.178+0000 7f4db020d640 1 -- 172.21.15.3:0/4202080816 shutdown_connections 2024-03-25T18:33:00.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:00.178+0000 7f4db020d640 1 -- 172.21.15.3:0/4202080816 >> 172.21.15.3:0/4202080816 conn(0x7f4da80617a0 msgr2=0x7f4da8140e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:00.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:00.178+0000 7f4db020d640 1 -- 172.21.15.3:0/4202080816 shutdown_connections 2024-03-25T18:33:00.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:00.178+0000 7f4db020d640 1 -- 172.21.15.3:0/4202080816 wait complete. 2024-03-25T18:33:00.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:00.179+0000 7f4db020d640 1 librados: shutdown 2024-03-25T18:33:00.192 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:02.801 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:08:16.530 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:02.860 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:08:16.589 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:02.897 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:08:16.626 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:02.909 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:08:16.638 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:02.917 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:08:16.646 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:02.922 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:08:16.615 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:02.933 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:08:16.625 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:02.943 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:08:16.672 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:02.960 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:08:16.652 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:06.999 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:07.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.114+0000 7f783cd82640 10 monclient: get_monmap_and_config 2024-03-25T18:33:07.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.114+0000 7f783cd82640 10 monclient: build_initial_monmap 2024-03-25T18:33:07.115 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.114+0000 7f783cd82640 10 monclient: monmap: 2024-03-25T18:33:07.115 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:33:07.115 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:07.115 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:33:07.114271+0000 2024-03-25T18:33:07.115 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:33:07.114271+0000 2024-03-25T18:33:07.115 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:07.115 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:07.115 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:33:07.115 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:33:07.115 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:33:07.115 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:07.116 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.115+0000 7f783cd82640 1 Processor -- start 2024-03-25T18:33:07.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.115+0000 7f783cd82640 1 -- start start 2024-03-25T18:33:07.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.115+0000 7f783cd82640 10 monclient: init 2024-03-25T18:33:07.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.115+0000 7f783cd82640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:07.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.115+0000 7f783cd82640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:33:07.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.115+0000 7f783cd82640 10 monclient(hunting): picked mon.noname-c con 0x7f78380b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:33:07.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.115+0000 7f783cd82640 10 monclient(hunting): picked mon.noname-b con 0x7f78380b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:33:07.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.116+0000 7f783cd82640 10 monclient(hunting): picked mon.noname-a con 0x7f783813c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:33:07.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.116+0000 7f783cd82640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78380ab0e0 con 0x7f783813c480 2024-03-25T18:33:07.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.116+0000 7f783cd82640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78380ab3c0 con 0x7f78380b4ff0 2024-03-25T18:33:07.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.116+0000 7f783cd82640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78380a9f30 con 0x7f78380b4670 2024-03-25T18:33:07.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.116+0000 7f783cd82640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:07.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.116+0000 7f783cd82640 10 monclient(hunting): authenticate will time out at 7841.375401s 2024-03-25T18:33:07.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.120+0000 7f7836d9d640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f783813c480 0x7f783814dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58474/0 (socket says 172.21.15.3:58474) 2024-03-25T18:33:07.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.120+0000 7f7836d9d640 1 -- 172.21.15.3:0/397264635 learned_addr learned my addr 172.21.15.3:0/397264635 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:33:07.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.138+0000 7f7835d9b640 1 -- 172.21.15.3:0/397264635 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296976152 0 0) 0x7f78380ab0e0 con 0x7f783813c480 2024-03-25T18:33:07.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.138+0000 7f7835d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:33:07.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.138+0000 7f7835d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:07.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.139+0000 7f7835d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:07.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.139+0000 7f7835d9b640 10 monclient(hunting): my global_id is 4596 2024-03-25T18:33:07.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.139+0000 7f7835d9b640 1 -- 172.21.15.3:0/397264635 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7824003520 con 0x7f783813c480 2024-03-25T18:33:07.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.140+0000 7f7835d9b640 1 -- 172.21.15.3:0/397264635 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 591050874 0 0) 0x7f78380a9f30 con 0x7f78380b4670 2024-03-25T18:33:07.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.140+0000 7f7835d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:33:07.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.140+0000 7f7835d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:07.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.140+0000 7f7835d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:07.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.140+0000 7f7835d9b640 10 monclient(hunting): my global_id is 4553 2024-03-25T18:33:07.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.140+0000 7f7835d9b640 1 -- 172.21.15.3:0/397264635 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78380ab0e0 con 0x7f78380b4670 2024-03-25T18:33:07.156 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.155+0000 7f7835d9b640 1 -- 172.21.15.3:0/397264635 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 401817401 0 0) 0x7f78380ab3c0 con 0x7f78380b4ff0 2024-03-25T18:33:07.156 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.155+0000 7f7835d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:33:07.156 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.155+0000 7f7835d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:07.156 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.155+0000 7f7835d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:07.156 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.155+0000 7f7835d9b640 10 monclient(hunting): my global_id is 4549 2024-03-25T18:33:07.156 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.155+0000 7f7835d9b640 1 -- 172.21.15.3:0/397264635 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78380a9f30 con 0x7f78380b4ff0 2024-03-25T18:33:07.161 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.161+0000 7f7835d9b640 1 -- 172.21.15.3:0/397264635 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 449060826 0 0) 0x7f7824003520 con 0x7f783813c480 2024-03-25T18:33:07.161 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.161+0000 7f7835d9b640 1 -- 172.21.15.3:0/397264635 >> v1:172.21.15.3:6790/0 conn(0x7f78380b4670 legacy=0x7f78381425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:07.161 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.161+0000 7f7835d9b640 1 -- 172.21.15.3:0/397264635 >> v1:172.21.15.16:6789/0 conn(0x7f78380b4ff0 legacy=0x7f783813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:07.161 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.161+0000 7f7835d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:33:07.161 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.161+0000 7f7835d9b640 1 monclient: found mon.noname-a 2024-03-25T18:33:07.161 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.161+0000 7f7835d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:07.161 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.161+0000 7f7835d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:33:07.162 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.161+0000 7f7835d9b640 1 -- 172.21.15.3:0/397264635 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78380a9b70 con 0x7f783813c480 2024-03-25T18:33:07.162 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.161+0000 7f7835d9b640 10 monclient: _finish_auth 0 2024-03-25T18:33:07.162 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.161+0000 7f7835d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:33:07.162 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.161+0000 7f7835d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:07.162 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.161+0000 7f7835d9b640 1 -- 172.21.15.3:0/397264635 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7830001a10 con 0x7f783813c480 2024-03-25T18:33:07.162 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.161+0000 7f7835d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:07.162 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.161+0000 7f7835d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:07.162 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.161+0000 7f7835d9b640 10 monclient: dump: 2024-03-25T18:33:07.162 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:07.162 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:07.162 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:07.163 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:07.163 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:07.163 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:07.163 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:07.163 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:07.163 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:07.163 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:07.163 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.161+0000 7f783cd82640 5 monclient: authenticate success, global_id 4596 2024-03-25T18:33:07.163 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.161+0000 7f783cd82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:07.170 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.170+0000 7f7835d9b640 1 -- 172.21.15.3:0/397264635 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7830001b70 con 0x7f783813c480 2024-03-25T18:33:07.171 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.170+0000 7f7835d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:07.171 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.170+0000 7f7835d9b640 1 -- 172.21.15.3:0/397264635 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7830001f80 con 0x7f783813c480 2024-03-25T18:33:07.171 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.170+0000 7f7835d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:07.171 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.170+0000 7f7835d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:07.171 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.170+0000 7f7835d9b640 10 monclient: dump: 2024-03-25T18:33:07.171 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:07.171 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:07.171 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:07.171 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:07.172 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:07.172 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:07.172 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:07.172 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:07.172 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:07.172 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:07.172 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.170+0000 7f783cd82640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:07.172 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.170+0000 7f783cd82640 4 set_mon_vals no callback set 2024-03-25T18:33:07.173 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.171+0000 7f783cd82640 10 monclient: shutdown 2024-03-25T18:33:07.173 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.171+0000 7f783cd82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:07.173 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.171+0000 7f783cd82640 1 -- 172.21.15.3:0/397264635 >> v1:172.21.15.3:6789/0 conn(0x7f783813c480 legacy=0x7f783814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:07.173 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.171+0000 7f783cd82640 1 -- 172.21.15.3:0/397264635 shutdown_connections 2024-03-25T18:33:07.173 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.171+0000 7f783cd82640 1 -- 172.21.15.3:0/397264635 >> 172.21.15.3:0/397264635 conn(0x7f78380617a0 msgr2=0x7f7838061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:07.173 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.171+0000 7f783cd82640 1 -- 172.21.15.3:0/397264635 shutdown_connections 2024-03-25T18:33:07.174 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.171+0000 7f783cd82640 1 -- 172.21.15.3:0/397264635 wait complete. 2024-03-25T18:33:07.174 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.172+0000 7f783cd82640 10 monclient: build_initial_monmap 2024-03-25T18:33:07.174 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.172+0000 7f783cd82640 10 monclient: monmap: 2024-03-25T18:33:07.174 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:33:07.174 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:07.174 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:33:07.174 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:33:07.174 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:07.175 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:33:07.175 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:33:07.175 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:33:07.175 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:33:07.175 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:07.175 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.173+0000 7f783cd82640 1 librados: starting msgr at 2024-03-25T18:33:07.175 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.173+0000 7f783cd82640 1 librados: starting objecter 2024-03-25T18:33:07.175 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.173+0000 7f783cd82640 1 Processor -- start 2024-03-25T18:33:07.175 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.173+0000 7f783cd82640 1 -- start start 2024-03-25T18:33:07.175 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.173+0000 7f783cd82640 1 librados: setting wanted keys 2024-03-25T18:33:07.175 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.173+0000 7f783cd82640 1 librados: calling monclient init 2024-03-25T18:33:07.176 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.173+0000 7f783cd82640 10 monclient: init 2024-03-25T18:33:07.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.174+0000 7f783cd82640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:07.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.174+0000 7f783cd82640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:33:07.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.174+0000 7f783cd82640 10 monclient(hunting): picked mon.noname-b con 0x7f78380b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:33:07.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.174+0000 7f783cd82640 10 monclient(hunting): picked mon.noname-a con 0x7f78380b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:33:07.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.174+0000 7f783cd82640 10 monclient(hunting): picked mon.noname-c con 0x7f783813c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:33:07.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.174+0000 7f783cd82640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7838071f70 con 0x7f78380b4ff0 2024-03-25T18:33:07.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.174+0000 7f783cd82640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78380abfb0 con 0x7f78380b4670 2024-03-25T18:33:07.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.174+0000 7f783cd82640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78380ac290 con 0x7f783813c480 2024-03-25T18:33:07.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.174+0000 7f783cd82640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:07.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.174+0000 7f783cd82640 10 monclient(hunting): authenticate will time out at 7841.433596s 2024-03-25T18:33:07.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.180+0000 7f7836d9d640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f783813c480 0x7f78381e6640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:58960/0 (socket says 172.21.15.3:58960) 2024-03-25T18:33:07.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.180+0000 7f7836d9d640 1 -- 172.21.15.3:0/3705309898 learned_addr learned my addr 172.21.15.3:0/3705309898 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:33:07.182 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.182+0000 7f782effd640 10 client.?.objecter ms_handle_connect 0x7f783813c480 2024-03-25T18:33:07.182 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.182+0000 7f782effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:07.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.188+0000 7f782effd640 1 -- 172.21.15.3:0/3705309898 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 755833608 0 0) 0x7f78380ac290 con 0x7f783813c480 2024-03-25T18:33:07.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.188+0000 7f782effd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:07.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.188+0000 7f782effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:07.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.188+0000 7f782effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:07.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.188+0000 7f782effd640 10 monclient(hunting): my global_id is 4556 2024-03-25T18:33:07.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.188+0000 7f782effd640 1 -- 172.21.15.3:0/3705309898 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7818004eb0 con 0x7f783813c480 2024-03-25T18:33:07.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.193+0000 7f782effd640 10 client.?.objecter ms_handle_connect 0x7f78380b4670 2024-03-25T18:33:07.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.193+0000 7f782effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:07.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.196+0000 7f782effd640 10 client.?.objecter ms_handle_connect 0x7f78380b4ff0 2024-03-25T18:33:07.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.196+0000 7f782effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:07.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.200+0000 7f782effd640 1 -- 172.21.15.3:0/3705309898 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3329244726 0 0) 0x7f7818004eb0 con 0x7f783813c480 2024-03-25T18:33:07.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.200+0000 7f782effd640 1 -- 172.21.15.3:0/3705309898 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78380ac290 con 0x7f783813c480 2024-03-25T18:33:07.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.200+0000 7f782effd640 1 -- 172.21.15.3:0/3705309898 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7830001ec0 con 0x7f783813c480 2024-03-25T18:33:07.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.200+0000 7f782effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:07.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.200+0000 7f782effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:07.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.200+0000 7f782effd640 10 monclient(hunting): dump: 2024-03-25T18:33:07.201 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:07.201 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:07.201 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:07.201 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:07.201 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:07.201 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:07.201 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:07.201 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:07.201 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:07.201 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:07.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.200+0000 7f782effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:07.202 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.202+0000 7f782effd640 1 -- 172.21.15.3:0/3705309898 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2256089261 0 0) 0x7f78380abfb0 con 0x7f78380b4670 2024-03-25T18:33:07.202 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.202+0000 7f782effd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:07.202 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.202+0000 7f782effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:07.202 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.202+0000 7f782effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:07.202 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.202+0000 7f782effd640 10 monclient(hunting): my global_id is 4552 2024-03-25T18:33:07.202 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.202+0000 7f782effd640 1 -- 172.21.15.3:0/3705309898 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7818004eb0 con 0x7f78380b4670 2024-03-25T18:33:07.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.203+0000 7f782effd640 1 -- 172.21.15.3:0/3705309898 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2295699609 0 0) 0x7f7838071f70 con 0x7f78380b4ff0 2024-03-25T18:33:07.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.204+0000 7f782effd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:07.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.204+0000 7f782effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:07.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.204+0000 7f782effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:07.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.204+0000 7f782effd640 10 monclient(hunting): my global_id is 4599 2024-03-25T18:33:07.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.204+0000 7f782effd640 1 -- 172.21.15.3:0/3705309898 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78380abfb0 con 0x7f78380b4ff0 2024-03-25T18:33:07.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.215+0000 7f782effd640 1 -- 172.21.15.3:0/3705309898 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2401049902 0 0) 0x7f78380ac290 con 0x7f783813c480 2024-03-25T18:33:07.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.215+0000 7f782effd640 1 -- 172.21.15.3:0/3705309898 >> v1:172.21.15.16:6789/0 conn(0x7f78380b4670 legacy=0x7f7838071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:07.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.215+0000 7f782effd640 1 -- 172.21.15.3:0/3705309898 >> v1:172.21.15.3:6789/0 conn(0x7f78380b4ff0 legacy=0x7f78380ac990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:07.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.216+0000 7f782effd640 10 monclient: _finish_hunting 0 2024-03-25T18:33:07.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.216+0000 7f782effd640 1 monclient: found mon.c 2024-03-25T18:33:07.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.216+0000 7f782effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:07.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.216+0000 7f782effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:33:07.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.216+0000 7f782effd640 1 -- 172.21.15.3:0/3705309898 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7838071880 con 0x7f783813c480 2024-03-25T18:33:07.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.216+0000 7f782effd640 10 monclient: _finish_auth 0 2024-03-25T18:33:07.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.216+0000 7f782effd640 10 monclient: _check_auth_tickets 2024-03-25T18:33:07.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.216+0000 7f782effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:07.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.216+0000 7f783cd82640 5 monclient: authenticate success, global_id 4556 2024-03-25T18:33:07.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.216+0000 7f783cd82640 10 monclient: _renew_subs 2024-03-25T18:33:07.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.216+0000 7f783cd82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:33:07.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.216+0000 7f783cd82640 1 -- 172.21.15.3:0/3705309898 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7838071b20 con 0x7f783813c480 2024-03-25T18:33:07.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.216+0000 7f783cd82640 10 client.4556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:07.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.216+0000 7f783cd82640 10 monclient: _renew_subs 2024-03-25T18:33:07.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.216+0000 7f783cd82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:33:07.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.216+0000 7f783cd82640 1 -- 172.21.15.3:0/3705309898 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f78380ad280 con 0x7f783813c480 2024-03-25T18:33:07.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.216+0000 7f783cd82640 1 librados: init done 2024-03-25T18:33:07.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.217+0000 7f783cd82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:07.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.217+0000 7f783cd82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:07.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.217+0000 7f783cd82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:33:07.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.217+0000 7f783cd82640 1 -- 172.21.15.3:0/3705309898 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f783806dbd0 con 0x7f783813c480 2024-03-25T18:33:07.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.556+0000 7f782effd640 1 -- 172.21.15.3:0/3705309898 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7830004e10 con 0x7f783813c480 2024-03-25T18:33:07.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.556+0000 7f782effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:07.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.556+0000 7f782effd640 1 -- 172.21.15.3:0/3705309898 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7830005120 con 0x7f783813c480 2024-03-25T18:33:07.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.556+0000 7f782effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:07.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.556+0000 7f782effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:07.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.556+0000 7f782effd640 10 monclient: dump: 2024-03-25T18:33:07.557 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:07.557 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:07.557 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:07.557 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:07.557 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:07.557 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:07.557 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:07.557 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:07.557 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:07.558 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:07.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.556+0000 7f783559a640 4 set_mon_vals no callback set 2024-03-25T18:33:07.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.557+0000 7f782effd640 1 -- 172.21.15.3:0/3705309898 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f783001b0f0 con 0x7f783813c480 2024-03-25T18:33:07.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.557+0000 7f783659c640 10 monclient: get_auth_request con 0x7f781806dea0 auth_method 0 2024-03-25T18:33:07.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.557+0000 7f782effd640 10 client.4556.objecter ms_handle_connect 0x7f781806dea0 2024-03-25T18:33:07.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.563+0000 7f782effd640 1 -- 172.21.15.3:0/3705309898 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f7830054be0 con 0x7f783813c480 2024-03-25T18:33:07.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.563+0000 7f782effd640 10 client.4556.objecter ms_dispatch 0x7f7838000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:07.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.563+0000 7f782effd640 3 client.4556.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:07.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.563+0000 7f782effd640 3 client.4556.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:07.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.563+0000 7f782effd640 20 client.4556.objecter dump_active .. 0 homeless 2024-03-25T18:33:07.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.563+0000 7f782effd640 1 -- 172.21.15.3:0/3705309898 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f78300506f0 con 0x7f783813c480 2024-03-25T18:33:07.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.563+0000 7f782effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:07.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.563+0000 7f782effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:07.817 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.817+0000 7f783cd82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:07.817 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.817+0000 7f783cd82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:07.817 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.817+0000 7f783cd82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:33:07.817 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.817+0000 7f783cd82640 1 -- 172.21.15.3:0/3705309898 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7838071cb0 con 0x7f783813c480 2024-03-25T18:33:07.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.953+0000 7f782effd640 1 -- 172.21.15.3:0/3705309898 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f7838071cb0 con 0x7f783813c480 2024-03-25T18:33:07.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.954+0000 7f782effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:07.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.954+0000 7f782effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:07.954 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:33:07.955 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:07.956 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.956+0000 7f783cd82640 10 librados: watch_flush enter 2024-03-25T18:33:07.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.956+0000 7f783cd82640 10 librados: watch_flush exit 2024-03-25T18:33:07.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.956+0000 7f783cd82640 20 client.4556.objecter shutdown clearing up homeless session... 2024-03-25T18:33:07.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.956+0000 7f783cd82640 10 client.4556.objecter successfully canceled tick 2024-03-25T18:33:07.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.956+0000 7f783cd82640 1 -- 172.21.15.3:0/3705309898 >> v1:172.21.15.3:6800/4195140966 conn(0x7f781806dea0 legacy=0x7f7818070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:07.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.956+0000 7f783cd82640 10 monclient: shutdown 2024-03-25T18:33:07.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.956+0000 7f783cd82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:07.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.956+0000 7f783cd82640 1 -- 172.21.15.3:0/3705309898 >> v1:172.21.15.3:6790/0 conn(0x7f783813c480 legacy=0x7f78381e6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:07.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.956+0000 7f783cd82640 1 -- 172.21.15.3:0/3705309898 shutdown_connections 2024-03-25T18:33:07.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.956+0000 7f783cd82640 1 -- 172.21.15.3:0/3705309898 >> 172.21.15.3:0/3705309898 conn(0x7f78380617a0 msgr2=0x7f7838145670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:07.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.956+0000 7f783cd82640 1 -- 172.21.15.3:0/3705309898 shutdown_connections 2024-03-25T18:33:07.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.956+0000 7f783cd82640 1 -- 172.21.15.3:0/3705309898 wait complete. 2024-03-25T18:33:07.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:07.956+0000 7f783cd82640 1 librados: shutdown 2024-03-25T18:33:07.968 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:14.775 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:14.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.888+0000 7f8fdf2b3640 10 monclient: get_monmap_and_config 2024-03-25T18:33:14.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.888+0000 7f8fdf2b3640 10 monclient: build_initial_monmap 2024-03-25T18:33:14.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.888+0000 7f8fdf2b3640 10 monclient: monmap: 2024-03-25T18:33:14.890 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:33:14.890 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:14.890 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:33:14.889369+0000 2024-03-25T18:33:14.890 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:33:14.889369+0000 2024-03-25T18:33:14.890 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:14.890 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:14.890 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:33:14.890 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:33:14.891 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:33:14.891 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:14.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.888+0000 7f8fdf2b3640 1 Processor -- start 2024-03-25T18:33:14.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.889+0000 7f8fdf2b3640 1 -- start start 2024-03-25T18:33:14.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.889+0000 7f8fdf2b3640 10 monclient: init 2024-03-25T18:33:14.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.889+0000 7f8fdf2b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:14.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.889+0000 7f8fdf2b3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:33:14.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.889+0000 7f8fdf2b3640 10 monclient(hunting): picked mon.noname-c con 0x7f8fd80b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:33:14.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.890+0000 7f8fdf2b3640 10 monclient(hunting): picked mon.noname-b con 0x7f8fd80b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:33:14.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.890+0000 7f8fdf2b3640 10 monclient(hunting): picked mon.noname-a con 0x7f8fd813c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:33:14.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.890+0000 7f8fdf2b3640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd80ab0e0 con 0x7f8fd813c480 2024-03-25T18:33:14.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.890+0000 7f8fdf2b3640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd80ab3c0 con 0x7f8fd80b4ff0 2024-03-25T18:33:14.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.890+0000 7f8fdf2b3640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd80a9f30 con 0x7f8fd80b4670 2024-03-25T18:33:14.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.890+0000 7f8fdf2b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:14.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.890+0000 7f8fdf2b3640 10 monclient(hunting): authenticate will time out at 7849.150560s 2024-03-25T18:33:14.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.891+0000 7f8fdc84e640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f8fd80b4670 0x7f8fd81425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:52966/0 (socket says 172.21.15.3:52966) 2024-03-25T18:33:14.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.891+0000 7f8fdc84e640 1 -- 172.21.15.3:0/3171895786 learned_addr learned my addr 172.21.15.3:0/3171895786 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:33:14.909 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.908+0000 7f8fcffff640 1 -- 172.21.15.3:0/3171895786 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2625077009 0 0) 0x7f8fd80ab0e0 con 0x7f8fd813c480 2024-03-25T18:33:14.909 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.908+0000 7f8fcffff640 10 monclient(hunting): _negotiate 2024-03-25T18:33:14.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.908+0000 7f8fcffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:14.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.908+0000 7f8fcffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:14.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.908+0000 7f8fcffff640 10 monclient(hunting): my global_id is 4602 2024-03-25T18:33:14.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.908+0000 7f8fcffff640 1 -- 172.21.15.3:0/3171895786 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fc4003520 con 0x7f8fd813c480 2024-03-25T18:33:14.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.921+0000 7f8fcffff640 1 -- 172.21.15.3:0/3171895786 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3938186736 0 0) 0x7f8fc4003520 con 0x7f8fd813c480 2024-03-25T18:33:14.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.921+0000 7f8fcffff640 1 -- 172.21.15.3:0/3171895786 >> v1:172.21.15.3:6790/0 conn(0x7f8fd80b4670 legacy=0x7f8fd81425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:14.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.921+0000 7f8fcffff640 1 -- 172.21.15.3:0/3171895786 >> v1:172.21.15.16:6789/0 conn(0x7f8fd80b4ff0 legacy=0x7f8fd813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:14.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.921+0000 7f8fcffff640 10 monclient: _finish_hunting 0 2024-03-25T18:33:14.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.921+0000 7f8fcffff640 1 monclient: found mon.noname-a 2024-03-25T18:33:14.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.921+0000 7f8fcffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:14.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.921+0000 7f8fcffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:33:14.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.921+0000 7f8fcffff640 1 -- 172.21.15.3:0/3171895786 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd80a9b70 con 0x7f8fd813c480 2024-03-25T18:33:14.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.921+0000 7f8fcffff640 10 monclient: _finish_auth 0 2024-03-25T18:33:14.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.921+0000 7f8fcffff640 10 monclient: _check_auth_tickets 2024-03-25T18:33:14.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.921+0000 7f8fcffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:14.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.921+0000 7f8fcffff640 1 -- 172.21.15.3:0/3171895786 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f8fd0001a10 con 0x7f8fd813c480 2024-03-25T18:33:14.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.921+0000 7f8fcffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:14.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.921+0000 7f8fcffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:14.924 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.921+0000 7f8fcffff640 10 monclient: dump: 2024-03-25T18:33:14.924 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:14.924 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:14.924 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:14.924 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:14.924 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:14.924 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:14.924 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:14.924 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:14.924 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:14.924 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:14.924 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.921+0000 7f8fdf2b3640 5 monclient: authenticate success, global_id 4602 2024-03-25T18:33:14.925 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.921+0000 7f8fdf2b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:14.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.986+0000 7f8fcffff640 1 -- 172.21.15.3:0/3171895786 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8fd0001b70 con 0x7f8fd813c480 2024-03-25T18:33:14.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.986+0000 7f8fcffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:14.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.986+0000 7f8fcffff640 1 -- 172.21.15.3:0/3171895786 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f8fd0001f80 con 0x7f8fd813c480 2024-03-25T18:33:14.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.986+0000 7f8fdf2b3640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:14.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.986+0000 7f8fdf2b3640 4 set_mon_vals no callback set 2024-03-25T18:33:14.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.986+0000 7f8fcffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:14.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.986+0000 7f8fcffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:14.989 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.986+0000 7f8fcffff640 10 monclient: dump: 2024-03-25T18:33:14.989 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:14.989 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:14.989 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:14.989 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:14.989 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:14.989 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:14.989 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:14.989 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:14.989 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:14.989 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:14.990 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.987+0000 7f8fdf2b3640 10 monclient: shutdown 2024-03-25T18:33:14.990 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.987+0000 7f8fdf2b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:14.990 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.987+0000 7f8fdf2b3640 1 -- 172.21.15.3:0/3171895786 >> v1:172.21.15.3:6789/0 conn(0x7f8fd813c480 legacy=0x7f8fd814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:14.990 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.987+0000 7f8fdf2b3640 1 -- 172.21.15.3:0/3171895786 shutdown_connections 2024-03-25T18:33:14.990 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.987+0000 7f8fdf2b3640 1 -- 172.21.15.3:0/3171895786 >> 172.21.15.3:0/3171895786 conn(0x7f8fd80617a0 msgr2=0x7f8fd8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:14.990 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.987+0000 7f8fdf2b3640 1 -- 172.21.15.3:0/3171895786 shutdown_connections 2024-03-25T18:33:14.990 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.987+0000 7f8fdf2b3640 1 -- 172.21.15.3:0/3171895786 wait complete. 2024-03-25T18:33:14.992 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.988+0000 7f8fdf2b3640 10 monclient: build_initial_monmap 2024-03-25T18:33:14.992 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.988+0000 7f8fdf2b3640 10 monclient: monmap: 2024-03-25T18:33:14.992 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:33:14.992 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:14.992 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:33:14.993 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:33:14.993 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:14.993 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:33:14.993 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:33:14.993 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:33:14.993 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:33:14.993 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:14.993 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.988+0000 7f8fdf2b3640 1 librados: starting msgr at 2024-03-25T18:33:14.993 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.988+0000 7f8fdf2b3640 1 librados: starting objecter 2024-03-25T18:33:14.993 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.988+0000 7f8fdf2b3640 1 Processor -- start 2024-03-25T18:33:14.993 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.988+0000 7f8fdf2b3640 1 -- start start 2024-03-25T18:33:14.993 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.988+0000 7f8fdf2b3640 1 librados: setting wanted keys 2024-03-25T18:33:14.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.988+0000 7f8fdf2b3640 1 librados: calling monclient init 2024-03-25T18:33:14.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.988+0000 7f8fdf2b3640 10 monclient: init 2024-03-25T18:33:14.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.989+0000 7f8fdf2b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:14.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.989+0000 7f8fdf2b3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:33:14.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.989+0000 7f8fdf2b3640 10 monclient(hunting): picked mon.noname-a con 0x7f8fd80b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:33:14.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.989+0000 7f8fdf2b3640 10 monclient(hunting): picked mon.noname-b con 0x7f8fd80b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:33:14.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.989+0000 7f8fdf2b3640 10 monclient(hunting): picked mon.noname-c con 0x7f8fd813c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:33:14.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.989+0000 7f8fdf2b3640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd80aec60 con 0x7f8fd80b4670 2024-03-25T18:33:14.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.990+0000 7f8fdf2b3640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd80b0800 con 0x7f8fd80b4ff0 2024-03-25T18:33:14.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.990+0000 7f8fdf2b3640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd80b0ae0 con 0x7f8fd813c480 2024-03-25T18:33:14.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.990+0000 7f8fdf2b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:14.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.990+0000 7f8fdf2b3640 10 monclient(hunting): authenticate will time out at 7849.250385s 2024-03-25T18:33:14.997 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.996+0000 7f8fdc84e640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8fd80b4ff0 0x7f8fd80af800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:52966/0 (socket says 172.21.15.3:52966) 2024-03-25T18:33:14.997 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.996+0000 7f8fdc84e640 1 -- 172.21.15.3:0/2228958365 learned_addr learned my addr 172.21.15.3:0/2228958365 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:33:14.998 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.997+0000 7f8fcdffb640 10 client.?.objecter ms_handle_connect 0x7f8fd80b4ff0 2024-03-25T18:33:14.998 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:14.997+0000 7f8fcdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:15.009 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.008+0000 7f8fcdffb640 10 client.?.objecter ms_handle_connect 0x7f8fd80b4670 2024-03-25T18:33:15.009 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.008+0000 7f8fcdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:15.013 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.011+0000 7f8fcdffb640 1 -- 172.21.15.3:0/2228958365 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 586503528 0 0) 0x7f8fd80b0800 con 0x7f8fd80b4ff0 2024-03-25T18:33:15.013 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.011+0000 7f8fcdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:33:15.013 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.011+0000 7f8fcdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:15.013 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.011+0000 7f8fcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:15.013 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.011+0000 7f8fcdffb640 10 monclient(hunting): my global_id is 4555 2024-03-25T18:33:15.013 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.012+0000 7f8fcdffb640 1 -- 172.21.15.3:0/2228958365 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fbc004eb0 con 0x7f8fd80b4ff0 2024-03-25T18:33:15.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.018+0000 7f8fcdffb640 1 -- 172.21.15.3:0/2228958365 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4047683200 0 0) 0x7f8fd80aec60 con 0x7f8fd80b4670 2024-03-25T18:33:15.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.018+0000 7f8fcdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:33:15.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.018+0000 7f8fcdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:15.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.018+0000 7f8fcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:15.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.018+0000 7f8fcdffb640 10 monclient(hunting): my global_id is 4605 2024-03-25T18:33:15.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.018+0000 7f8fcdffb640 1 -- 172.21.15.3:0/2228958365 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fd80b0800 con 0x7f8fd80b4670 2024-03-25T18:33:15.025 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.024+0000 7f8fcdffb640 1 -- 172.21.15.3:0/2228958365 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4274000762 0 0) 0x7f8fbc004eb0 con 0x7f8fd80b4ff0 2024-03-25T18:33:15.025 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.024+0000 7f8fcdffb640 1 -- 172.21.15.3:0/2228958365 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fd80aec60 con 0x7f8fd80b4ff0 2024-03-25T18:33:15.025 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.024+0000 7f8fcdffb640 1 -- 172.21.15.3:0/2228958365 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f8fc80018b0 con 0x7f8fd80b4ff0 2024-03-25T18:33:15.025 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.024+0000 7f8fcdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:15.026 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.024+0000 7f8fcdffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:15.026 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.024+0000 7f8fcdffb640 10 monclient(hunting): dump: 2024-03-25T18:33:15.026 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:15.026 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:15.026 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:15.026 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:15.026 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:15.026 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:15.026 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:15.026 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:15.026 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:15.026 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:15.027 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.024+0000 7f8fcdffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:15.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.041+0000 7f8fcdffb640 1 -- 172.21.15.3:0/2228958365 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4018523905 0 0) 0x7f8fd80aec60 con 0x7f8fd80b4ff0 2024-03-25T18:33:15.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.041+0000 7f8fcdffb640 1 -- 172.21.15.3:0/2228958365 >> v1:172.21.15.3:6790/0 conn(0x7f8fd813c480 legacy=0x7f8fd81e2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:15.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.041+0000 7f8fcdffb640 1 -- 172.21.15.3:0/2228958365 >> v1:172.21.15.3:6789/0 conn(0x7f8fd80b4670 legacy=0x7f8fd80af0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:15.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.041+0000 7f8fcdffb640 10 monclient: _finish_hunting 0 2024-03-25T18:33:15.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.041+0000 7f8fcdffb640 1 monclient: found mon.b 2024-03-25T18:33:15.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.041+0000 7f8fcdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:15.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.041+0000 7f8fcdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:33:15.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.041+0000 7f8fcdffb640 1 -- 172.21.15.3:0/2228958365 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd80b0dc0 con 0x7f8fd80b4ff0 2024-03-25T18:33:15.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.041+0000 7f8fcdffb640 10 monclient: _finish_auth 0 2024-03-25T18:33:15.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.041+0000 7f8fcdffb640 10 monclient: _check_auth_tickets 2024-03-25T18:33:15.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.041+0000 7f8fcdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:15.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.041+0000 7f8fdf2b3640 5 monclient: authenticate success, global_id 4555 2024-03-25T18:33:15.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.041+0000 7f8fdf2b3640 10 monclient: _renew_subs 2024-03-25T18:33:15.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.041+0000 7f8fdf2b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:33:15.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.041+0000 7f8fdf2b3640 1 -- 172.21.15.3:0/2228958365 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fd80ae6f0 con 0x7f8fd80b4ff0 2024-03-25T18:33:15.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.041+0000 7f8fdf2b3640 10 client.4555.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:15.045 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.041+0000 7f8fdf2b3640 10 monclient: _renew_subs 2024-03-25T18:33:15.045 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.041+0000 7f8fdf2b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:33:15.045 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.041+0000 7f8fdf2b3640 1 -- 172.21.15.3:0/2228958365 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd80b1730 con 0x7f8fd80b4ff0 2024-03-25T18:33:15.045 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.041+0000 7f8fdf2b3640 1 librados: init done 2024-03-25T18:33:15.045 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.042+0000 7f8fdf2b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:15.045 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.042+0000 7f8fdf2b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:15.045 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.042+0000 7f8fdf2b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:33:15.045 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.042+0000 7f8fdf2b3640 1 -- 172.21.15.3:0/2228958365 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd80aea40 con 0x7f8fd80b4ff0 2024-03-25T18:33:15.337 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.335+0000 7f8fcdffb640 1 -- 172.21.15.3:0/2228958365 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8fc80047e0 con 0x7f8fd80b4ff0 2024-03-25T18:33:15.337 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.335+0000 7f8fcdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:15.337 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.336+0000 7f8fcdffb640 1 -- 172.21.15.3:0/2228958365 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f8fc8004c30 con 0x7f8fd80b4ff0 2024-03-25T18:33:15.337 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.336+0000 7f8fcdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:15.337 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.336+0000 7f8fcdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:15.337 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.336+0000 7f8fcdffb640 10 monclient: dump: 2024-03-25T18:33:15.338 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:15.338 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:15.338 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:15.338 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:15.338 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:15.338 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:15.338 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:15.338 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:15.342 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:15.342 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:15.342 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.336+0000 7f8fcf7fe640 4 set_mon_vals no callback set 2024-03-25T18:33:15.343 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.336+0000 7f8fcdffb640 1 -- 172.21.15.3:0/2228958365 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f8fc8004e50 con 0x7f8fd80b4ff0 2024-03-25T18:33:15.343 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.337+0000 7f8fdd04f640 10 monclient: get_auth_request con 0x7f8fbc06dcb0 auth_method 0 2024-03-25T18:33:15.343 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.337+0000 7f8fcdffb640 10 client.4555.objecter ms_handle_connect 0x7f8fbc06dcb0 2024-03-25T18:33:15.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.342+0000 7f8fcdffb640 1 -- 172.21.15.3:0/2228958365 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f8fc80547c0 con 0x7f8fd80b4ff0 2024-03-25T18:33:15.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.342+0000 7f8fcdffb640 10 client.4555.objecter ms_dispatch 0x7f8fd8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:15.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.342+0000 7f8fcdffb640 3 client.4555.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:15.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.342+0000 7f8fcdffb640 3 client.4555.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:15.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.343+0000 7f8fcdffb640 20 client.4555.objecter dump_active .. 0 homeless 2024-03-25T18:33:15.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.343+0000 7f8fcdffb640 1 -- 172.21.15.3:0/2228958365 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8fc80502d0 con 0x7f8fd80b4ff0 2024-03-25T18:33:15.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.343+0000 7f8fcdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:15.345 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.343+0000 7f8fcdffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:15.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.599+0000 7f8fdf2b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:15.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.599+0000 7f8fdf2b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:15.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.599+0000 7f8fdf2b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:33:15.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.599+0000 7f8fdf2b3640 1 -- 172.21.15.3:0/2228958365 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8fd806dbd0 con 0x7f8fd80b4ff0 2024-03-25T18:33:15.623 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.621+0000 7f8fcdffb640 1 -- 172.21.15.3:0/2228958365 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f8fc8052200 con 0x7f8fd80b4ff0 2024-03-25T18:33:15.623 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.621+0000 7f8fcdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:15.623 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.621+0000 7f8fcdffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:15.623 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:33:15.623 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:15.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.624+0000 7f8fdf2b3640 10 librados: watch_flush enter 2024-03-25T18:33:15.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.624+0000 7f8fdf2b3640 10 librados: watch_flush exit 2024-03-25T18:33:15.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.624+0000 7f8fdf2b3640 20 client.4555.objecter shutdown clearing up homeless session... 2024-03-25T18:33:15.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.624+0000 7f8fdf2b3640 10 client.4555.objecter successfully canceled tick 2024-03-25T18:33:15.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.624+0000 7f8fdf2b3640 1 -- 172.21.15.3:0/2228958365 >> v1:172.21.15.3:6800/4195140966 conn(0x7f8fbc06dcb0 legacy=0x7f8fbc070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:15.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.624+0000 7f8fdf2b3640 10 monclient: shutdown 2024-03-25T18:33:15.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.624+0000 7f8fdf2b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:15.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.624+0000 7f8fdf2b3640 1 -- 172.21.15.3:0/2228958365 >> v1:172.21.15.16:6789/0 conn(0x7f8fd80b4ff0 legacy=0x7f8fd80af800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:15.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.624+0000 7f8fdf2b3640 1 -- 172.21.15.3:0/2228958365 shutdown_connections 2024-03-25T18:33:15.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.624+0000 7f8fdf2b3640 1 -- 172.21.15.3:0/2228958365 >> 172.21.15.3:0/2228958365 conn(0x7f8fd80617a0 msgr2=0x7f8fd81407f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:15.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.625+0000 7f8fdf2b3640 1 -- 172.21.15.3:0/2228958365 shutdown_connections 2024-03-25T18:33:15.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.625+0000 7f8fdf2b3640 1 -- 172.21.15.3:0/2228958365 wait complete. 2024-03-25T18:33:15.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:15.625+0000 7f8fdf2b3640 1 librados: shutdown 2024-03-25T18:33:15.636 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:22.442 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:22.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.555+0000 7f0f94b6d640 10 monclient: get_monmap_and_config 2024-03-25T18:33:22.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.555+0000 7f0f94b6d640 10 monclient: build_initial_monmap 2024-03-25T18:33:22.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.555+0000 7f0f94b6d640 10 monclient: monmap: 2024-03-25T18:33:22.556 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:33:22.556 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:22.556 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:33:22.556517+0000 2024-03-25T18:33:22.556 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:33:22.556517+0000 2024-03-25T18:33:22.557 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:22.557 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:22.557 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:33:22.557 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:33:22.557 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:33:22.557 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:22.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.556+0000 7f0f94b6d640 1 Processor -- start 2024-03-25T18:33:22.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.556+0000 7f0f94b6d640 1 -- start start 2024-03-25T18:33:22.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.556+0000 7f0f94b6d640 10 monclient: init 2024-03-25T18:33:22.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.557+0000 7f0f94b6d640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:22.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.557+0000 7f0f94b6d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:33:22.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.557+0000 7f0f94b6d640 10 monclient(hunting): picked mon.noname-b con 0x7f0f900b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:33:22.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.557+0000 7f0f94b6d640 10 monclient(hunting): picked mon.noname-c con 0x7f0f900b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:33:22.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.557+0000 7f0f94b6d640 10 monclient(hunting): picked mon.noname-a con 0x7f0f9013c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:33:22.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.557+0000 7f0f94b6d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f900ab0e0 con 0x7f0f9013c480 2024-03-25T18:33:22.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.557+0000 7f0f94b6d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f900ab3c0 con 0x7f0f900b4670 2024-03-25T18:33:22.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.557+0000 7f0f94b6d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f900a9f30 con 0x7f0f900b4ff0 2024-03-25T18:33:22.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.557+0000 7f0f94b6d640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:22.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.557+0000 7f0f94b6d640 10 monclient(hunting): authenticate will time out at 7856.817633s 2024-03-25T18:33:22.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.575+0000 7f0f87fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0f900b4670 0x7f0f901425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:38270/0 (socket says 172.21.15.3:38270) 2024-03-25T18:33:22.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.575+0000 7f0f87fff640 1 -- 172.21.15.3:0/2807599409 learned_addr learned my addr 172.21.15.3:0/2807599409 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:33:22.596 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.595+0000 7f0f8dd9b640 1 -- 172.21.15.3:0/2807599409 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1171273451 0 0) 0x7f0f900a9f30 con 0x7f0f900b4ff0 2024-03-25T18:33:22.597 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.595+0000 7f0f8dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:33:22.597 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.595+0000 7f0f8dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:22.597 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.595+0000 7f0f8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:22.597 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.595+0000 7f0f8dd9b640 10 monclient(hunting): my global_id is 4559 2024-03-25T18:33:22.597 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.595+0000 7f0f8dd9b640 1 -- 172.21.15.3:0/2807599409 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f7c003520 con 0x7f0f900b4ff0 2024-03-25T18:33:22.598 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.597+0000 7f0f8dd9b640 1 -- 172.21.15.3:0/2807599409 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3407664028 0 0) 0x7f0f900ab3c0 con 0x7f0f900b4670 2024-03-25T18:33:22.598 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.597+0000 7f0f8dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:33:22.598 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.597+0000 7f0f8dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:22.598 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.597+0000 7f0f8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:22.598 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.597+0000 7f0f8dd9b640 10 monclient(hunting): my global_id is 4558 2024-03-25T18:33:22.598 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.597+0000 7f0f8dd9b640 1 -- 172.21.15.3:0/2807599409 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f900a9f30 con 0x7f0f900b4670 2024-03-25T18:33:22.605 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.604+0000 7f0f8dd9b640 1 -- 172.21.15.3:0/2807599409 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3034916022 0 0) 0x7f0f900ab0e0 con 0x7f0f9013c480 2024-03-25T18:33:22.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.604+0000 7f0f8dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:33:22.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.604+0000 7f0f8dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:22.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.604+0000 7f0f8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:22.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.604+0000 7f0f8dd9b640 10 monclient(hunting): my global_id is 4608 2024-03-25T18:33:22.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.604+0000 7f0f8dd9b640 1 -- 172.21.15.3:0/2807599409 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f900ab3c0 con 0x7f0f9013c480 2024-03-25T18:33:22.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.613+0000 7f0f8dd9b640 1 -- 172.21.15.3:0/2807599409 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1368030809 0 0) 0x7f0f900a9f30 con 0x7f0f900b4670 2024-03-25T18:33:22.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.613+0000 7f0f8dd9b640 1 -- 172.21.15.3:0/2807599409 >> v1:172.21.15.3:6790/0 conn(0x7f0f900b4ff0 legacy=0x7f0f9013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:22.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.613+0000 7f0f8dd9b640 1 -- 172.21.15.3:0/2807599409 >> v1:172.21.15.3:6789/0 conn(0x7f0f9013c480 legacy=0x7f0f9014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:22.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.613+0000 7f0f8dd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:33:22.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.613+0000 7f0f8dd9b640 1 monclient: found mon.noname-c 2024-03-25T18:33:22.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.613+0000 7f0f8dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:22.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.613+0000 7f0f8dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:33:22.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.613+0000 7f0f8dd9b640 1 -- 172.21.15.3:0/2807599409 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f900a9b70 con 0x7f0f900b4670 2024-03-25T18:33:22.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.613+0000 7f0f8dd9b640 10 monclient: _finish_auth 0 2024-03-25T18:33:22.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.613+0000 7f0f8dd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:33:22.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.613+0000 7f0f8dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:22.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.613+0000 7f0f8dd9b640 1 -- 172.21.15.3:0/2807599409 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f0f78001670 con 0x7f0f900b4670 2024-03-25T18:33:22.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.613+0000 7f0f94b6d640 5 monclient: authenticate success, global_id 4558 2024-03-25T18:33:22.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.613+0000 7f0f8dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:22.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.613+0000 7f0f8dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:22.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.613+0000 7f0f8dd9b640 10 monclient: dump: 2024-03-25T18:33:22.615 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:22.615 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:22.615 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:22.615 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:22.615 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:22.616 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:22.616 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:22.616 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:22.616 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:22.616 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:22.616 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.613+0000 7f0f94b6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:22.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.624+0000 7f0f8dd9b640 1 -- 172.21.15.3:0/2807599409 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f780017d0 con 0x7f0f900b4670 2024-03-25T18:33:22.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.624+0000 7f0f8dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:22.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.624+0000 7f0f8dd9b640 1 -- 172.21.15.3:0/2807599409 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f0f78001be0 con 0x7f0f900b4670 2024-03-25T18:33:22.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.624+0000 7f0f8dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:22.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.624+0000 7f0f8dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:22.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.624+0000 7f0f8dd9b640 10 monclient: dump: 2024-03-25T18:33:22.626 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:22.626 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:22.626 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:22.626 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:22.626 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:22.626 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:22.626 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:22.626 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:22.626 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:22.626 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:22.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.624+0000 7f0f94b6d640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:22.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.624+0000 7f0f94b6d640 4 set_mon_vals no callback set 2024-03-25T18:33:22.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.625+0000 7f0f94b6d640 10 monclient: shutdown 2024-03-25T18:33:22.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.625+0000 7f0f94b6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:22.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.625+0000 7f0f94b6d640 1 -- 172.21.15.3:0/2807599409 >> v1:172.21.15.16:6789/0 conn(0x7f0f900b4670 legacy=0x7f0f901425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:22.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.625+0000 7f0f94b6d640 1 -- 172.21.15.3:0/2807599409 shutdown_connections 2024-03-25T18:33:22.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.625+0000 7f0f94b6d640 1 -- 172.21.15.3:0/2807599409 >> 172.21.15.3:0/2807599409 conn(0x7f0f900617a0 msgr2=0x7f0f90061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:22.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.625+0000 7f0f94b6d640 1 -- 172.21.15.3:0/2807599409 shutdown_connections 2024-03-25T18:33:22.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.625+0000 7f0f94b6d640 1 -- 172.21.15.3:0/2807599409 wait complete. 2024-03-25T18:33:22.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.626+0000 7f0f94b6d640 10 monclient: build_initial_monmap 2024-03-25T18:33:22.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.626+0000 7f0f94b6d640 10 monclient: monmap: 2024-03-25T18:33:22.628 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:33:22.628 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:22.628 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:33:22.628 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:33:22.629 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:22.629 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:33:22.629 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:33:22.629 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:33:22.629 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:33:22.629 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:22.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.626+0000 7f0f94b6d640 1 librados: starting msgr at 2024-03-25T18:33:22.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.626+0000 7f0f94b6d640 1 librados: starting objecter 2024-03-25T18:33:22.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.627+0000 7f0f94b6d640 1 Processor -- start 2024-03-25T18:33:22.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.627+0000 7f0f94b6d640 1 -- start start 2024-03-25T18:33:22.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.627+0000 7f0f94b6d640 1 librados: setting wanted keys 2024-03-25T18:33:22.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.627+0000 7f0f94b6d640 1 librados: calling monclient init 2024-03-25T18:33:22.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.627+0000 7f0f94b6d640 10 monclient: init 2024-03-25T18:33:22.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.627+0000 7f0f94b6d640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:22.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.627+0000 7f0f94b6d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:33:22.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.627+0000 7f0f94b6d640 10 monclient(hunting): picked mon.noname-c con 0x7f0f900b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:33:22.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.628+0000 7f0f94b6d640 10 monclient(hunting): picked mon.noname-b con 0x7f0f900b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:33:22.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.628+0000 7f0f94b6d640 10 monclient(hunting): picked mon.noname-a con 0x7f0f9013c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:33:22.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.628+0000 7f0f94b6d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f90144360 con 0x7f0f9013c480 2024-03-25T18:33:22.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.628+0000 7f0f94b6d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f90143410 con 0x7f0f900b4ff0 2024-03-25T18:33:22.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.628+0000 7f0f94b6d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f901436f0 con 0x7f0f900b4670 2024-03-25T18:33:22.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.628+0000 7f0f94b6d640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:22.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.628+0000 7f0f94b6d640 10 monclient(hunting): authenticate will time out at 7856.888505s 2024-03-25T18:33:22.636 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.635+0000 7f0f87fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0f900b4ff0 0x7f0f901db130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:38286/0 (socket says 172.21.15.3:38286) 2024-03-25T18:33:22.636 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.635+0000 7f0f87fff640 1 -- 172.21.15.3:0/1643284353 learned_addr learned my addr 172.21.15.3:0/1643284353 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:33:22.643 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.642+0000 7f0f86ffd640 10 client.?.objecter ms_handle_connect 0x7f0f900b4ff0 2024-03-25T18:33:22.643 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.642+0000 7f0f86ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:22.646 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.646+0000 7f0f86ffd640 10 client.?.objecter ms_handle_connect 0x7f0f900b4670 2024-03-25T18:33:22.647 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.646+0000 7f0f86ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:22.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.650+0000 7f0f86ffd640 10 client.?.objecter ms_handle_connect 0x7f0f9013c480 2024-03-25T18:33:22.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.650+0000 7f0f86ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:22.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.653+0000 7f0f86ffd640 1 -- 172.21.15.3:0/1643284353 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2940213120 0 0) 0x7f0f901436f0 con 0x7f0f900b4670 2024-03-25T18:33:22.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.653+0000 7f0f86ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:22.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.653+0000 7f0f86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:22.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.653+0000 7f0f86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:22.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.653+0000 7f0f86ffd640 10 monclient(hunting): my global_id is 4562 2024-03-25T18:33:22.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.654+0000 7f0f86ffd640 1 -- 172.21.15.3:0/1643284353 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f70004eb0 con 0x7f0f900b4670 2024-03-25T18:33:22.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.655+0000 7f0f86ffd640 1 -- 172.21.15.3:0/1643284353 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2690842738 0 0) 0x7f0f90144360 con 0x7f0f9013c480 2024-03-25T18:33:22.656 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.655+0000 7f0f86ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:22.656 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.655+0000 7f0f86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:22.656 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.655+0000 7f0f86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:22.656 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.655+0000 7f0f86ffd640 10 monclient(hunting): my global_id is 4611 2024-03-25T18:33:22.656 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.655+0000 7f0f86ffd640 1 -- 172.21.15.3:0/1643284353 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f901436f0 con 0x7f0f9013c480 2024-03-25T18:33:22.656 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.655+0000 7f0f86ffd640 1 -- 172.21.15.3:0/1643284353 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2526648817 0 0) 0x7f0f90143410 con 0x7f0f900b4ff0 2024-03-25T18:33:22.656 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.655+0000 7f0f86ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:22.656 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.655+0000 7f0f86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:22.657 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.655+0000 7f0f86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:22.657 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.655+0000 7f0f86ffd640 10 monclient(hunting): my global_id is 4561 2024-03-25T18:33:22.657 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.655+0000 7f0f86ffd640 1 -- 172.21.15.3:0/1643284353 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f90144360 con 0x7f0f900b4ff0 2024-03-25T18:33:22.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.664+0000 7f0f86ffd640 1 -- 172.21.15.3:0/1643284353 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3596496864 0 0) 0x7f0f901436f0 con 0x7f0f9013c480 2024-03-25T18:33:22.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.665+0000 7f0f86ffd640 1 -- 172.21.15.3:0/1643284353 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f90143410 con 0x7f0f9013c480 2024-03-25T18:33:22.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.665+0000 7f0f86ffd640 1 -- 172.21.15.3:0/1643284353 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f0f88001b50 con 0x7f0f9013c480 2024-03-25T18:33:22.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.665+0000 7f0f86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:22.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.665+0000 7f0f86ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:22.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.665+0000 7f0f86ffd640 10 monclient(hunting): dump: 2024-03-25T18:33:22.666 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:22.666 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:22.666 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:22.666 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:22.666 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:22.666 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:22.667 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:22.667 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:22.667 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:22.667 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:22.667 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.665+0000 7f0f86ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:22.667 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.666+0000 7f0f86ffd640 1 -- 172.21.15.3:0/1643284353 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3192742502 0 0) 0x7f0f90144360 con 0x7f0f900b4ff0 2024-03-25T18:33:22.667 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.666+0000 7f0f86ffd640 1 -- 172.21.15.3:0/1643284353 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f901436f0 con 0x7f0f900b4ff0 2024-03-25T18:33:22.667 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.666+0000 7f0f86ffd640 1 -- 172.21.15.3:0/1643284353 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f0f78001b20 con 0x7f0f900b4ff0 2024-03-25T18:33:22.667 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.666+0000 7f0f86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:22.668 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.666+0000 7f0f86ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:22.668 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.666+0000 7f0f86ffd640 10 monclient(hunting): dump: 2024-03-25T18:33:22.668 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:22.668 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:22.668 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:22.668 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:22.668 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:22.668 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:22.668 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:22.668 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:22.668 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:22.668 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:22.672 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.671+0000 7f0f86ffd640 1 -- 172.21.15.3:0/1643284353 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1541731818 0 0) 0x7f0f901436f0 con 0x7f0f900b4ff0 2024-03-25T18:33:22.672 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.671+0000 7f0f86ffd640 1 -- 172.21.15.3:0/1643284353 >> v1:172.21.15.3:6790/0 conn(0x7f0f900b4670 legacy=0x7f0f90144c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:22.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.671+0000 7f0f86ffd640 1 -- 172.21.15.3:0/1643284353 >> v1:172.21.15.3:6789/0 conn(0x7f0f9013c480 legacy=0x7f0f901db840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:22.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.671+0000 7f0f86ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:33:22.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.671+0000 7f0f86ffd640 1 monclient: found mon.b 2024-03-25T18:33:22.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.671+0000 7f0f86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:22.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.671+0000 7f0f86ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:33:22.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.671+0000 7f0f86ffd640 1 -- 172.21.15.3:0/1643284353 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f901dbf50 con 0x7f0f900b4ff0 2024-03-25T18:33:22.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.671+0000 7f0f86ffd640 10 monclient: _finish_auth 0 2024-03-25T18:33:22.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.671+0000 7f0f86ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:33:22.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.671+0000 7f0f86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:22.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.671+0000 7f0f94b6d640 5 monclient: authenticate success, global_id 4561 2024-03-25T18:33:22.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.671+0000 7f0f94b6d640 10 monclient: _renew_subs 2024-03-25T18:33:22.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.671+0000 7f0f94b6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:33:22.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.671+0000 7f0f94b6d640 1 -- 172.21.15.3:0/1643284353 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f901445f0 con 0x7f0f900b4ff0 2024-03-25T18:33:22.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.671+0000 7f0f94b6d640 10 client.4561.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:22.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.671+0000 7f0f94b6d640 10 monclient: _renew_subs 2024-03-25T18:33:22.674 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.671+0000 7f0f94b6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:33:22.674 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.671+0000 7f0f94b6d640 1 -- 172.21.15.3:0/1643284353 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f90145580 con 0x7f0f900b4ff0 2024-03-25T18:33:22.674 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.671+0000 7f0f94b6d640 1 librados: init done 2024-03-25T18:33:22.674 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.672+0000 7f0f94b6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:22.674 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.672+0000 7f0f94b6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:22.674 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.672+0000 7f0f94b6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:33:22.674 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.672+0000 7f0f94b6d640 1 -- 172.21.15.3:0/1643284353 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f9006dbd0 con 0x7f0f900b4ff0 2024-03-25T18:33:22.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.958+0000 7f0f86ffd640 1 -- 172.21.15.3:0/1643284353 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f78004a90 con 0x7f0f900b4ff0 2024-03-25T18:33:22.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.958+0000 7f0f86ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:22.960 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.958+0000 7f0f86ffd640 1 -- 172.21.15.3:0/1643284353 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f0f78004da0 con 0x7f0f900b4ff0 2024-03-25T18:33:22.960 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.958+0000 7f0f86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:22.960 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.958+0000 7f0f86ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:22.960 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.958+0000 7f0f8d59a640 4 set_mon_vals no callback set 2024-03-25T18:33:22.960 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.958+0000 7f0f86ffd640 10 monclient: dump: 2024-03-25T18:33:22.960 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:22.960 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:22.960 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:22.960 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:22.960 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:22.960 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:22.960 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:22.960 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:22.960 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:22.960 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:22.961 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.959+0000 7f0f86ffd640 1 -- 172.21.15.3:0/1643284353 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f0f7801ad70 con 0x7f0f900b4ff0 2024-03-25T18:33:22.961 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.960+0000 7f0f8e59c640 10 monclient: get_auth_request con 0x7f0f7006dfd0 auth_method 0 2024-03-25T18:33:22.961 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.960+0000 7f0f86ffd640 10 client.4561.objecter ms_handle_connect 0x7f0f7006dfd0 2024-03-25T18:33:22.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.969+0000 7f0f86ffd640 1 -- 172.21.15.3:0/1643284353 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f0f78054890 con 0x7f0f900b4ff0 2024-03-25T18:33:22.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.969+0000 7f0f86ffd640 10 client.4561.objecter ms_dispatch 0x7f0f90000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:22.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.969+0000 7f0f86ffd640 3 client.4561.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:22.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.969+0000 7f0f86ffd640 3 client.4561.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:22.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.969+0000 7f0f86ffd640 20 client.4561.objecter dump_active .. 0 homeless 2024-03-25T18:33:22.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.970+0000 7f0f86ffd640 1 -- 172.21.15.3:0/1643284353 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0f780501d0 con 0x7f0f900b4ff0 2024-03-25T18:33:22.971 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.970+0000 7f0f86ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:22.971 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:22.970+0000 7f0f86ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:23.128 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:08:36.857 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:23.196 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:08:36.925 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:23.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:23.199+0000 7f0f94b6d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:23.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:23.199+0000 7f0f94b6d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:23.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:23.199+0000 7f0f94b6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:33:23.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:23.199+0000 7f0f94b6d640 1 -- 172.21.15.3:0/1643284353 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f901447d0 con 0x7f0f900b4ff0 2024-03-25T18:33:23.223 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:08:36.952 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:23.256 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:08:36.948 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:23.259 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:08:36.989 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:23.267 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:08:36.996 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:23.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:23.277+0000 7f0f86ffd640 1 -- 172.21.15.3:0/1643284353 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f0f780522d0 con 0x7f0f900b4ff0 2024-03-25T18:33:23.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:23.277+0000 7f0f86ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:23.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:23.277+0000 7f0f86ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:23.279 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:33:23.279 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:23.279 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:08:36.971 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:23.281 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:23.279+0000 7f0f94b6d640 10 librados: watch_flush enter 2024-03-25T18:33:23.281 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:23.279+0000 7f0f94b6d640 10 librados: watch_flush exit 2024-03-25T18:33:23.281 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:23.279+0000 7f0f94b6d640 20 client.4561.objecter shutdown clearing up homeless session... 2024-03-25T18:33:23.281 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:23.279+0000 7f0f94b6d640 10 client.4561.objecter successfully canceled tick 2024-03-25T18:33:23.281 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:23.279+0000 7f0f94b6d640 1 -- 172.21.15.3:0/1643284353 >> v1:172.21.15.3:6800/4195140966 conn(0x7f0f7006dfd0 legacy=0x7f0f70070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:23.281 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:23.279+0000 7f0f94b6d640 10 monclient: shutdown 2024-03-25T18:33:23.281 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:23.279+0000 7f0f94b6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:23.281 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:23.279+0000 7f0f94b6d640 1 -- 172.21.15.3:0/1643284353 >> v1:172.21.15.16:6789/0 conn(0x7f0f900b4ff0 legacy=0x7f0f901db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:23.282 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:23.279+0000 7f0f94b6d640 1 -- 172.21.15.3:0/1643284353 shutdown_connections 2024-03-25T18:33:23.282 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:23.279+0000 7f0f94b6d640 1 -- 172.21.15.3:0/1643284353 >> 172.21.15.3:0/1643284353 conn(0x7f0f900617a0 msgr2=0x7f0f90141b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:23.282 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:23.279+0000 7f0f94b6d640 1 -- 172.21.15.3:0/1643284353 shutdown_connections 2024-03-25T18:33:23.282 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:23.279+0000 7f0f94b6d640 1 -- 172.21.15.3:0/1643284353 wait complete. 2024-03-25T18:33:23.282 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:23.280+0000 7f0f94b6d640 1 librados: shutdown 2024-03-25T18:33:23.291 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:23.296 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:08:37.025 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:23.302 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:08:36.994 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:30.098 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:30.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.211+0000 7fe32cf8f640 10 monclient: get_monmap_and_config 2024-03-25T18:33:30.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.211+0000 7fe32cf8f640 10 monclient: build_initial_monmap 2024-03-25T18:33:30.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.211+0000 7fe32cf8f640 10 monclient: monmap: 2024-03-25T18:33:30.213 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:33:30.213 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:30.213 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:33:30.212969+0000 2024-03-25T18:33:30.213 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:33:30.212969+0000 2024-03-25T18:33:30.213 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:30.213 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:30.213 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:33:30.213 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:33:30.214 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:33:30.214 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:30.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.212+0000 7fe32cf8f640 1 Processor -- start 2024-03-25T18:33:30.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.212+0000 7fe32cf8f640 1 -- start start 2024-03-25T18:33:30.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.212+0000 7fe32cf8f640 10 monclient: init 2024-03-25T18:33:30.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.213+0000 7fe32cf8f640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:30.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.213+0000 7fe32cf8f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:33:30.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.213+0000 7fe32cf8f640 10 monclient(hunting): picked mon.noname-b con 0x7fe3280b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:33:30.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.213+0000 7fe32cf8f640 10 monclient(hunting): picked mon.noname-a con 0x7fe3280b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:33:30.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.213+0000 7fe32cf8f640 10 monclient(hunting): picked mon.noname-c con 0x7fe32813c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:33:30.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.213+0000 7fe32cf8f640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3280ab0e0 con 0x7fe3280b4670 2024-03-25T18:33:30.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.213+0000 7fe32cf8f640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3280ab3c0 con 0x7fe32813c480 2024-03-25T18:33:30.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.213+0000 7fe32cf8f640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3280a9f30 con 0x7fe3280b4ff0 2024-03-25T18:33:30.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.213+0000 7fe32cf8f640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:30.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.213+0000 7fe32cf8f640 10 monclient(hunting): authenticate will time out at 7864.474044s 2024-03-25T18:33:30.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.217+0000 7fe32659c640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fe3280b4ff0 0x7fe32813bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:55348/0 (socket says 172.21.15.3:55348) 2024-03-25T18:33:30.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.217+0000 7fe32659c640 1 -- 172.21.15.3:0/4084335202 learned_addr learned my addr 172.21.15.3:0/4084335202 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:33:30.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.225+0000 7fe325d9b640 1 -- 172.21.15.3:0/4084335202 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3416978226 0 0) 0x7fe3280a9f30 con 0x7fe3280b4ff0 2024-03-25T18:33:30.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.225+0000 7fe325d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:33:30.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.225+0000 7fe325d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:30.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.225+0000 7fe325d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:30.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.225+0000 7fe325d9b640 10 monclient(hunting): my global_id is 4565 2024-03-25T18:33:30.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.225+0000 7fe325d9b640 1 -- 172.21.15.3:0/4084335202 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe314003520 con 0x7fe3280b4ff0 2024-03-25T18:33:30.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.231+0000 7fe325d9b640 1 -- 172.21.15.3:0/4084335202 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1702265725 0 0) 0x7fe314003520 con 0x7fe3280b4ff0 2024-03-25T18:33:30.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.231+0000 7fe325d9b640 1 -- 172.21.15.3:0/4084335202 >> v1:172.21.15.16:6789/0 conn(0x7fe32813c480 legacy=0x7fe32814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:30.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.231+0000 7fe325d9b640 1 -- 172.21.15.3:0/4084335202 >> v1:172.21.15.3:6789/0 conn(0x7fe3280b4670 legacy=0x7fe3281425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:33:30.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.231+0000 7fe325d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:33:30.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.231+0000 7fe325d9b640 1 monclient: found mon.noname-b 2024-03-25T18:33:30.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.231+0000 7fe325d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:30.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.231+0000 7fe325d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-25T18:33:30.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.231+0000 7fe325d9b640 1 -- 172.21.15.3:0/4084335202 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3280a9b70 con 0x7fe3280b4ff0 2024-03-25T18:33:30.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.231+0000 7fe325d9b640 10 monclient: _finish_auth 0 2024-03-25T18:33:30.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.231+0000 7fe325d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:33:30.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.231+0000 7fe325d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:30.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.231+0000 7fe325d9b640 1 -- 172.21.15.3:0/4084335202 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fe318001690 con 0x7fe3280b4ff0 2024-03-25T18:33:30.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.231+0000 7fe325d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:30.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.231+0000 7fe325d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:30.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.231+0000 7fe325d9b640 10 monclient: dump: 2024-03-25T18:33:30.234 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:30.234 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:30.234 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:30.234 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:30.234 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:30.234 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:30.234 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:30.234 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:30.234 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:30.234 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:30.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.231+0000 7fe32cf8f640 5 monclient: authenticate success, global_id 4565 2024-03-25T18:33:30.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.231+0000 7fe32cf8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:30.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.239+0000 7fe325d9b640 1 -- 172.21.15.3:0/4084335202 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe318001860 con 0x7fe3280b4ff0 2024-03-25T18:33:30.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.239+0000 7fe325d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:30.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.239+0000 7fe325d9b640 1 -- 172.21.15.3:0/4084335202 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fe318001cb0 con 0x7fe3280b4ff0 2024-03-25T18:33:30.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.239+0000 7fe32cf8f640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:30.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.239+0000 7fe32cf8f640 4 set_mon_vals no callback set 2024-03-25T18:33:30.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.239+0000 7fe325d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:30.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.239+0000 7fe325d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:30.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.239+0000 7fe325d9b640 10 monclient: dump: 2024-03-25T18:33:30.241 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:30.241 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:30.241 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:30.241 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:30.241 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:30.241 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:30.241 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:30.241 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:30.241 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:30.241 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:30.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.240+0000 7fe32cf8f640 10 monclient: shutdown 2024-03-25T18:33:30.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.240+0000 7fe32cf8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:30.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.240+0000 7fe32cf8f640 1 -- 172.21.15.3:0/4084335202 >> v1:172.21.15.3:6790/0 conn(0x7fe3280b4ff0 legacy=0x7fe32813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:30.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.240+0000 7fe32cf8f640 1 -- 172.21.15.3:0/4084335202 shutdown_connections 2024-03-25T18:33:30.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.240+0000 7fe32cf8f640 1 -- 172.21.15.3:0/4084335202 >> 172.21.15.3:0/4084335202 conn(0x7fe3280617a0 msgr2=0x7fe328061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:30.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.240+0000 7fe32cf8f640 1 -- 172.21.15.3:0/4084335202 shutdown_connections 2024-03-25T18:33:30.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.240+0000 7fe32cf8f640 1 -- 172.21.15.3:0/4084335202 wait complete. 2024-03-25T18:33:30.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.241+0000 7fe32cf8f640 10 monclient: build_initial_monmap 2024-03-25T18:33:30.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.241+0000 7fe32cf8f640 10 monclient: monmap: 2024-03-25T18:33:30.243 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:33:30.243 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:30.243 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:33:30.243 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:33:30.244 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:30.244 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:33:30.244 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:33:30.244 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:33:30.244 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:33:30.244 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:30.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.242+0000 7fe32cf8f640 1 librados: starting msgr at 2024-03-25T18:33:30.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.242+0000 7fe32cf8f640 1 librados: starting objecter 2024-03-25T18:33:30.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.242+0000 7fe32cf8f640 1 Processor -- start 2024-03-25T18:33:30.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.242+0000 7fe32cf8f640 1 -- start start 2024-03-25T18:33:30.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.242+0000 7fe32cf8f640 1 librados: setting wanted keys 2024-03-25T18:33:30.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.242+0000 7fe32cf8f640 1 librados: calling monclient init 2024-03-25T18:33:30.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.242+0000 7fe32cf8f640 10 monclient: init 2024-03-25T18:33:30.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.243+0000 7fe32cf8f640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:30.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.243+0000 7fe32cf8f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:33:30.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.243+0000 7fe32cf8f640 10 monclient(hunting): picked mon.noname-a con 0x7fe3280b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:33:30.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.243+0000 7fe32cf8f640 10 monclient(hunting): picked mon.noname-c con 0x7fe3280b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:33:30.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.243+0000 7fe32cf8f640 10 monclient(hunting): picked mon.noname-b con 0x7fe32813c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:33:30.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.243+0000 7fe32cf8f640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe328144540 con 0x7fe3280b4670 2024-03-25T18:33:30.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.243+0000 7fe32cf8f640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe328146350 con 0x7fe32813c480 2024-03-25T18:33:30.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.243+0000 7fe32cf8f640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe328144790 con 0x7fe3280b4ff0 2024-03-25T18:33:30.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.243+0000 7fe32cf8f640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:30.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.243+0000 7fe32cf8f640 10 monclient(hunting): authenticate will time out at 7864.504230s 2024-03-25T18:33:30.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.249+0000 7fe31dd9b640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fe3280b4ff0 0x7fe328145530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:55368/0 (socket says 172.21.15.3:55368) 2024-03-25T18:33:30.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.249+0000 7fe31dd9b640 1 -- 172.21.15.3:0/4231592616 learned_addr learned my addr 172.21.15.3:0/4231592616 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:33:30.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.251+0000 7fe31f7fe640 10 client.?.objecter ms_handle_connect 0x7fe3280b4ff0 2024-03-25T18:33:30.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.251+0000 7fe31f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:30.268 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.267+0000 7fe31f7fe640 10 client.?.objecter ms_handle_connect 0x7fe32813c480 2024-03-25T18:33:30.268 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.267+0000 7fe31f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:30.269 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.268+0000 7fe31f7fe640 1 -- 172.21.15.3:0/4231592616 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1936996651 0 0) 0x7fe328144790 con 0x7fe3280b4ff0 2024-03-25T18:33:30.269 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.268+0000 7fe31f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:33:30.269 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.268+0000 7fe31f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:30.269 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.268+0000 7fe31f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:30.270 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.268+0000 7fe31f7fe640 10 monclient(hunting): my global_id is 4568 2024-03-25T18:33:30.270 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.268+0000 7fe31f7fe640 1 -- 172.21.15.3:0/4231592616 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe308004eb0 con 0x7fe3280b4ff0 2024-03-25T18:33:30.274 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.273+0000 7fe31f7fe640 10 client.?.objecter ms_handle_connect 0x7fe3280b4670 2024-03-25T18:33:30.275 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.273+0000 7fe31f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:30.279 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.278+0000 7fe31f7fe640 1 -- 172.21.15.3:0/4231592616 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4246676663 0 0) 0x7fe308004eb0 con 0x7fe3280b4ff0 2024-03-25T18:33:30.279 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.278+0000 7fe31f7fe640 1 -- 172.21.15.3:0/4231592616 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe328144790 con 0x7fe3280b4ff0 2024-03-25T18:33:30.279 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.278+0000 7fe31f7fe640 1 -- 172.21.15.3:0/4231592616 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fe310004080 con 0x7fe3280b4ff0 2024-03-25T18:33:30.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.278+0000 7fe31f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:30.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.278+0000 7fe31f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:30.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.278+0000 7fe31f7fe640 10 monclient(hunting): dump: 2024-03-25T18:33:30.280 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:30.280 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:30.280 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:30.280 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:30.280 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:30.280 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:30.280 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:30.280 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:30.280 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:30.280 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:30.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.278+0000 7fe31f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:30.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.283+0000 7fe31f7fe640 1 -- 172.21.15.3:0/4231592616 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 462346008 0 0) 0x7fe328144540 con 0x7fe3280b4670 2024-03-25T18:33:30.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.283+0000 7fe31f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:33:30.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.283+0000 7fe31f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:30.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.283+0000 7fe31f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:30.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.283+0000 7fe31f7fe640 10 monclient(hunting): my global_id is 4614 2024-03-25T18:33:30.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.283+0000 7fe31f7fe640 1 -- 172.21.15.3:0/4231592616 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe308004eb0 con 0x7fe3280b4670 2024-03-25T18:33:30.298 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.296+0000 7fe31f7fe640 1 -- 172.21.15.3:0/4231592616 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 263547904 0 0) 0x7fe308004eb0 con 0x7fe3280b4670 2024-03-25T18:33:30.298 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.296+0000 7fe31f7fe640 1 -- 172.21.15.3:0/4231592616 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe328144540 con 0x7fe3280b4670 2024-03-25T18:33:30.298 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.296+0000 7fe31f7fe640 1 -- 172.21.15.3:0/4231592616 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fe3180019d0 con 0x7fe3280b4670 2024-03-25T18:33:30.298 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.296+0000 7fe31f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:30.298 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.296+0000 7fe31f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:30.298 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.296+0000 7fe31f7fe640 10 monclient(hunting): dump: 2024-03-25T18:33:30.298 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:30.298 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:30.298 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:30.298 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:30.299 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:30.299 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:30.299 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:30.299 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:30.299 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:30.299 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:30.299 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.297+0000 7fe31f7fe640 1 -- 172.21.15.3:0/4231592616 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3317466952 0 0) 0x7fe328146350 con 0x7fe32813c480 2024-03-25T18:33:30.299 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.297+0000 7fe31f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:33:30.299 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.297+0000 7fe31f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:30.299 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.297+0000 7fe31f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:30.299 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.297+0000 7fe31f7fe640 10 monclient(hunting): my global_id is 4564 2024-03-25T18:33:30.300 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.297+0000 7fe31f7fe640 1 -- 172.21.15.3:0/4231592616 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe308004eb0 con 0x7fe32813c480 2024-03-25T18:33:30.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.299+0000 7fe31f7fe640 1 -- 172.21.15.3:0/4231592616 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2671482905 0 0) 0x7fe328144790 con 0x7fe3280b4ff0 2024-03-25T18:33:30.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.299+0000 7fe31f7fe640 1 -- 172.21.15.3:0/4231592616 >> v1:172.21.15.16:6789/0 conn(0x7fe32813c480 legacy=0x7fe328145c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:30.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.299+0000 7fe31f7fe640 1 -- 172.21.15.3:0/4231592616 >> v1:172.21.15.3:6789/0 conn(0x7fe3280b4670 legacy=0x7fe3281dbb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:30.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.299+0000 7fe31f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:33:30.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.299+0000 7fe31f7fe640 1 monclient: found mon.c 2024-03-25T18:33:30.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.299+0000 7fe31f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:30.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.299+0000 7fe31f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:33:30.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.299+0000 7fe31f7fe640 1 -- 172.21.15.3:0/4231592616 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe328144a70 con 0x7fe3280b4ff0 2024-03-25T18:33:30.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.299+0000 7fe31f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:33:30.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.299+0000 7fe31f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:33:30.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.299+0000 7fe31f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:30.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.299+0000 7fe32cf8f640 5 monclient: authenticate success, global_id 4568 2024-03-25T18:33:30.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.299+0000 7fe32cf8f640 10 monclient: _renew_subs 2024-03-25T18:33:30.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.299+0000 7fe32cf8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:33:30.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.299+0000 7fe32cf8f640 1 -- 172.21.15.3:0/4231592616 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe328144d50 con 0x7fe3280b4ff0 2024-03-25T18:33:30.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.299+0000 7fe32cf8f640 10 client.4568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:30.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.299+0000 7fe32cf8f640 10 monclient: _renew_subs 2024-03-25T18:33:30.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.299+0000 7fe32cf8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:33:30.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.299+0000 7fe32cf8f640 1 -- 172.21.15.3:0/4231592616 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe328133a60 con 0x7fe3280b4ff0 2024-03-25T18:33:30.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.299+0000 7fe32cf8f640 1 librados: init done 2024-03-25T18:33:30.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.300+0000 7fe32cf8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:30.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.300+0000 7fe32cf8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:30.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.300+0000 7fe32cf8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:33:30.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.300+0000 7fe32cf8f640 1 -- 172.21.15.3:0/4231592616 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe328143610 con 0x7fe3280b4ff0 2024-03-25T18:33:30.598 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.597+0000 7fe31f7fe640 1 -- 172.21.15.3:0/4231592616 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe310000ed0 con 0x7fe3280b4ff0 2024-03-25T18:33:30.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.597+0000 7fe31f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:30.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.597+0000 7fe31f7fe640 1 -- 172.21.15.3:0/4231592616 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fe310001260 con 0x7fe3280b4ff0 2024-03-25T18:33:30.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.597+0000 7fe31f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:30.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.597+0000 7fe31f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:30.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.597+0000 7fe31f7fe640 10 monclient: dump: 2024-03-25T18:33:30.599 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:30.599 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:30.599 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:30.599 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:30.599 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:30.599 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:30.600 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:30.600 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:30.600 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:30.600 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:30.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.597+0000 7fe32559a640 4 set_mon_vals no callback set 2024-03-25T18:33:30.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.598+0000 7fe31f7fe640 1 -- 172.21.15.3:0/4231592616 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7fe310001480 con 0x7fe3280b4ff0 2024-03-25T18:33:30.601 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.599+0000 7fe32659c640 10 monclient: get_auth_request con 0x7fe30806dfb0 auth_method 0 2024-03-25T18:33:30.601 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.599+0000 7fe31f7fe640 10 client.4568.objecter ms_handle_connect 0x7fe30806dfb0 2024-03-25T18:33:30.605 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.604+0000 7fe31f7fe640 1 -- 172.21.15.3:0/4231592616 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7fe3100547e0 con 0x7fe3280b4ff0 2024-03-25T18:33:30.605 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.604+0000 7fe31f7fe640 10 client.4568.objecter ms_dispatch 0x7fe328000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:30.605 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.604+0000 7fe31f7fe640 3 client.4568.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:30.605 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.604+0000 7fe31f7fe640 3 client.4568.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:30.605 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.604+0000 7fe31f7fe640 20 client.4568.objecter dump_active .. 0 homeless 2024-03-25T18:33:30.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.604+0000 7fe31f7fe640 1 -- 172.21.15.3:0/4231592616 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe3100502f0 con 0x7fe3280b4ff0 2024-03-25T18:33:30.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.604+0000 7fe31f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:30.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.604+0000 7fe31f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:30.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.846+0000 7fe32cf8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:30.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.846+0000 7fe32cf8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:30.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.846+0000 7fe32cf8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:33:30.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.846+0000 7fe32cf8f640 1 -- 172.21.15.3:0/4231592616 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe32806dbd0 con 0x7fe3280b4ff0 2024-03-25T18:33:30.883 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.882+0000 7fe31f7fe640 1 -- 172.21.15.3:0/4231592616 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7fe310052220 con 0x7fe3280b4ff0 2024-03-25T18:33:30.883 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.882+0000 7fe31f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:30.883 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.882+0000 7fe31f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:30.884 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:33:30.884 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:30.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.884+0000 7fe32cf8f640 10 librados: watch_flush enter 2024-03-25T18:33:30.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.885+0000 7fe32cf8f640 10 librados: watch_flush exit 2024-03-25T18:33:30.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.885+0000 7fe32cf8f640 20 client.4568.objecter shutdown clearing up homeless session... 2024-03-25T18:33:30.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.885+0000 7fe32cf8f640 10 client.4568.objecter successfully canceled tick 2024-03-25T18:33:30.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.885+0000 7fe32cf8f640 1 -- 172.21.15.3:0/4231592616 >> v1:172.21.15.3:6800/4195140966 conn(0x7fe30806dfb0 legacy=0x7fe308070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:30.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.885+0000 7fe32cf8f640 10 monclient: shutdown 2024-03-25T18:33:30.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.885+0000 7fe32cf8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:30.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.885+0000 7fe32cf8f640 1 -- 172.21.15.3:0/4231592616 >> v1:172.21.15.3:6790/0 conn(0x7fe3280b4ff0 legacy=0x7fe328145530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:30.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.885+0000 7fe32cf8f640 1 -- 172.21.15.3:0/4231592616 shutdown_connections 2024-03-25T18:33:30.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.885+0000 7fe32cf8f640 1 -- 172.21.15.3:0/4231592616 >> 172.21.15.3:0/4231592616 conn(0x7fe3280617a0 msgr2=0x7fe3281405c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:30.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.885+0000 7fe32cf8f640 1 -- 172.21.15.3:0/4231592616 shutdown_connections 2024-03-25T18:33:30.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.885+0000 7fe32cf8f640 1 -- 172.21.15.3:0/4231592616 wait complete. 2024-03-25T18:33:30.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:30.885+0000 7fe32cf8f640 1 librados: shutdown 2024-03-25T18:33:30.898 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:37.705 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:37.801 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.800+0000 7f560171c640 10 monclient: get_monmap_and_config 2024-03-25T18:33:37.802 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.800+0000 7f560171c640 10 monclient: build_initial_monmap 2024-03-25T18:33:37.802 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.800+0000 7f560171c640 10 monclient: monmap: 2024-03-25T18:33:37.802 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:33:37.802 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:37.802 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:33:37.801894+0000 2024-03-25T18:33:37.802 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:33:37.801894+0000 2024-03-25T18:33:37.802 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:37.802 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:37.802 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:33:37.802 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:33:37.802 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:33:37.802 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:37.803 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.801+0000 7f560171c640 1 Processor -- start 2024-03-25T18:33:37.804 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.801+0000 7f560171c640 1 -- start start 2024-03-25T18:33:37.804 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.801+0000 7f560171c640 10 monclient: init 2024-03-25T18:33:37.804 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.802+0000 7f560171c640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:37.804 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.802+0000 7f560171c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:33:37.804 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.802+0000 7f560171c640 10 monclient(hunting): picked mon.noname-b con 0x7f55fc0b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:33:37.804 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.802+0000 7f560171c640 10 monclient(hunting): picked mon.noname-c con 0x7f55fc0b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:33:37.804 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.802+0000 7f560171c640 10 monclient(hunting): picked mon.noname-a con 0x7f55fc13c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:33:37.804 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.802+0000 7f560171c640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55fc0ab0e0 con 0x7f55fc13c480 2024-03-25T18:33:37.804 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.802+0000 7f560171c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55fc0ab3c0 con 0x7f55fc0b4670 2024-03-25T18:33:37.804 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.802+0000 7f560171c640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55fc0a9f30 con 0x7f55fc0b4ff0 2024-03-25T18:33:37.804 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.802+0000 7f560171c640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:37.804 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.802+0000 7f560171c640 10 monclient(hunting): authenticate will time out at 7872.062517s 2024-03-25T18:33:37.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.818+0000 7f55fb7fe640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f55fc13c480 0x7f55fc14dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:49498/0 (socket says 172.21.15.3:49498) 2024-03-25T18:33:37.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.818+0000 7f55fb7fe640 1 -- 172.21.15.3:0/3835027456 learned_addr learned my addr 172.21.15.3:0/3835027456 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:33:37.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.830+0000 7f55fa7fc640 1 -- 172.21.15.3:0/3835027456 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2572564567 0 0) 0x7f55fc0ab0e0 con 0x7f55fc13c480 2024-03-25T18:33:37.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.830+0000 7f55fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:33:37.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.830+0000 7f55fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:37.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.830+0000 7f55fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:37.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.830+0000 7f55fa7fc640 10 monclient(hunting): my global_id is 4617 2024-03-25T18:33:37.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.830+0000 7f55fa7fc640 1 -- 172.21.15.3:0/3835027456 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55e8003520 con 0x7f55fc13c480 2024-03-25T18:33:37.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.847+0000 7f55fa7fc640 1 -- 172.21.15.3:0/3835027456 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 160339516 0 0) 0x7f55e8003520 con 0x7f55fc13c480 2024-03-25T18:33:37.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.847+0000 7f55fa7fc640 1 -- 172.21.15.3:0/3835027456 >> v1:172.21.15.3:6790/0 conn(0x7f55fc0b4ff0 legacy=0x7f55fc13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:33:37.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.847+0000 7f55fa7fc640 1 -- 172.21.15.3:0/3835027456 >> v1:172.21.15.16:6789/0 conn(0x7f55fc0b4670 legacy=0x7f55fc1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:33:37.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.847+0000 7f55fa7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:33:37.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.847+0000 7f55fa7fc640 1 monclient: found mon.noname-a 2024-03-25T18:33:37.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.847+0000 7f55fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:37.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.847+0000 7f55fa7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:33:37.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.847+0000 7f55fa7fc640 1 -- 172.21.15.3:0/3835027456 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55fc0a9b70 con 0x7f55fc13c480 2024-03-25T18:33:37.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.847+0000 7f55fa7fc640 10 monclient: _finish_auth 0 2024-03-25T18:33:37.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.847+0000 7f55fa7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:33:37.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.847+0000 7f55fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:37.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.847+0000 7f55fa7fc640 1 -- 172.21.15.3:0/3835027456 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f55f4001a10 con 0x7f55fc13c480 2024-03-25T18:33:37.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.847+0000 7f55fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:37.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.847+0000 7f55fa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:37.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.847+0000 7f55fa7fc640 10 monclient: dump: 2024-03-25T18:33:37.849 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:37.849 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:37.849 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:37.849 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:37.849 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:37.849 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:37.850 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:37.850 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:37.850 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:37.850 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:37.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.847+0000 7f560171c640 5 monclient: authenticate success, global_id 4617 2024-03-25T18:33:37.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.847+0000 7f560171c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:37.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.870+0000 7f55fa7fc640 1 -- 172.21.15.3:0/3835027456 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55f4001be0 con 0x7f55fc13c480 2024-03-25T18:33:37.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.870+0000 7f55fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:37.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.870+0000 7f55fa7fc640 1 -- 172.21.15.3:0/3835027456 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f55f4002030 con 0x7f55fc13c480 2024-03-25T18:33:37.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.870+0000 7f55fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:37.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.870+0000 7f55fa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:37.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.870+0000 7f55fa7fc640 10 monclient: dump: 2024-03-25T18:33:37.871 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:37.871 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:37.871 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:37.871 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:37.871 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:37.871 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:37.872 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:37.872 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:37.872 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:37.872 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:37.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.870+0000 7f560171c640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:37.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.870+0000 7f560171c640 4 set_mon_vals no callback set 2024-03-25T18:33:37.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.870+0000 7f560171c640 10 monclient: shutdown 2024-03-25T18:33:37.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.870+0000 7f560171c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:37.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.870+0000 7f560171c640 1 -- 172.21.15.3:0/3835027456 >> v1:172.21.15.3:6789/0 conn(0x7f55fc13c480 legacy=0x7f55fc14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:37.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.871+0000 7f560171c640 1 -- 172.21.15.3:0/3835027456 shutdown_connections 2024-03-25T18:33:37.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.871+0000 7f560171c640 1 -- 172.21.15.3:0/3835027456 >> 172.21.15.3:0/3835027456 conn(0x7f55fc0617a0 msgr2=0x7f55fc061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:37.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.871+0000 7f560171c640 1 -- 172.21.15.3:0/3835027456 shutdown_connections 2024-03-25T18:33:37.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.871+0000 7f560171c640 1 -- 172.21.15.3:0/3835027456 wait complete. 2024-03-25T18:33:37.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.871+0000 7f560171c640 10 monclient: build_initial_monmap 2024-03-25T18:33:37.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.871+0000 7f560171c640 10 monclient: monmap: 2024-03-25T18:33:37.874 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:33:37.874 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:37.874 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:33:37.875 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:33:37.875 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:37.875 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:33:37.875 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:33:37.875 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:33:37.875 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:33:37.875 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:37.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.872+0000 7f560171c640 1 librados: starting msgr at 2024-03-25T18:33:37.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.872+0000 7f560171c640 1 librados: starting objecter 2024-03-25T18:33:37.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.872+0000 7f560171c640 1 Processor -- start 2024-03-25T18:33:37.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.872+0000 7f560171c640 1 -- start start 2024-03-25T18:33:37.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.872+0000 7f560171c640 1 librados: setting wanted keys 2024-03-25T18:33:37.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.872+0000 7f560171c640 1 librados: calling monclient init 2024-03-25T18:33:37.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.872+0000 7f560171c640 10 monclient: init 2024-03-25T18:33:37.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.872+0000 7f560171c640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:37.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.872+0000 7f560171c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:33:37.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.872+0000 7f560171c640 10 monclient(hunting): picked mon.noname-c con 0x7f55fc0b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:33:37.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.872+0000 7f560171c640 10 monclient(hunting): picked mon.noname-a con 0x7f55fc0b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:33:37.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.872+0000 7f560171c640 10 monclient(hunting): picked mon.noname-b con 0x7f55fc13c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:33:37.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.872+0000 7f560171c640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55fc136940 con 0x7f55fc0b4ff0 2024-03-25T18:33:37.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.872+0000 7f560171c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55fc136bb0 con 0x7f55fc13c480 2024-03-25T18:33:37.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.872+0000 7f560171c640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55fc13e840 con 0x7f55fc0b4670 2024-03-25T18:33:37.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.872+0000 7f560171c640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:37.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.872+0000 7f560171c640 10 monclient(hunting): authenticate will time out at 7872.133257s 2024-03-25T18:33:37.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.876+0000 7f55faffd640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f55fc0b4670 0x7f55fc1450a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:60048/0 (socket says 172.21.15.3:60048) 2024-03-25T18:33:37.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.876+0000 7f55faffd640 1 -- 172.21.15.3:0/2686207084 learned_addr learned my addr 172.21.15.3:0/2686207084 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:33:37.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.877+0000 7f55f3fff640 10 client.?.objecter ms_handle_connect 0x7f55fc0b4670 2024-03-25T18:33:37.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.877+0000 7f55f3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:37.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.879+0000 7f55f3fff640 10 client.?.objecter ms_handle_connect 0x7f55fc13c480 2024-03-25T18:33:37.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.879+0000 7f55f3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:37.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.886+0000 7f55f3fff640 1 -- 172.21.15.3:0/2686207084 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2580889814 0 0) 0x7f55fc13e840 con 0x7f55fc0b4670 2024-03-25T18:33:37.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.886+0000 7f55f3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:33:37.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.886+0000 7f55f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:37.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.886+0000 7f55f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:37.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.886+0000 7f55f3fff640 10 monclient(hunting): my global_id is 4571 2024-03-25T18:33:37.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.886+0000 7f55f3fff640 1 -- 172.21.15.3:0/2686207084 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55dc004eb0 con 0x7f55fc0b4670 2024-03-25T18:33:37.898 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.897+0000 7f55f3fff640 1 -- 172.21.15.3:0/2686207084 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1696769551 0 0) 0x7f55dc004eb0 con 0x7f55fc0b4670 2024-03-25T18:33:37.898 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.897+0000 7f55f3fff640 1 -- 172.21.15.3:0/2686207084 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55fc13e840 con 0x7f55fc0b4670 2024-03-25T18:33:37.898 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.897+0000 7f55f3fff640 1 -- 172.21.15.3:0/2686207084 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f55ec001960 con 0x7f55fc0b4670 2024-03-25T18:33:37.899 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.897+0000 7f55f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:37.899 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.897+0000 7f55f3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:37.899 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.897+0000 7f55f3fff640 10 monclient(hunting): dump: 2024-03-25T18:33:37.899 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:37.899 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:37.899 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:37.899 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:37.899 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:37.899 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:37.899 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:37.899 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:37.899 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:37.899 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:37.899 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.897+0000 7f55f3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:37.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.906+0000 7f55f3fff640 1 -- 172.21.15.3:0/2686207084 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1992868811 0 0) 0x7f55fc136bb0 con 0x7f55fc13c480 2024-03-25T18:33:37.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.906+0000 7f55f3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:33:37.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.906+0000 7f55f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:37.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.906+0000 7f55f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:37.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.906+0000 7f55f3fff640 10 monclient(hunting): my global_id is 4567 2024-03-25T18:33:37.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.906+0000 7f55f3fff640 1 -- 172.21.15.3:0/2686207084 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55dc004eb0 con 0x7f55fc13c480 2024-03-25T18:33:37.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.910+0000 7f55f3fff640 1 -- 172.21.15.3:0/2686207084 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3949650949 0 0) 0x7f55fc13e840 con 0x7f55fc0b4670 2024-03-25T18:33:37.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.910+0000 7f55f3fff640 1 -- 172.21.15.3:0/2686207084 >> v1:172.21.15.16:6789/0 conn(0x7f55fc13c480 legacy=0x7f55fc1e2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:37.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.910+0000 7f55f3fff640 1 -- 172.21.15.3:0/2686207084 >> v1:172.21.15.3:6789/0 conn(0x7f55fc0b4ff0 legacy=0x7f55fc1dac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:37.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.910+0000 7f55f3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:33:37.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.910+0000 7f55f3fff640 1 monclient: found mon.c 2024-03-25T18:33:37.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.910+0000 7f55f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:37.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.910+0000 7f55f3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:33:37.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.910+0000 7f55f3fff640 1 -- 172.21.15.3:0/2686207084 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55fc13e9d0 con 0x7f55fc0b4670 2024-03-25T18:33:37.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.910+0000 7f55f3fff640 10 monclient: _finish_auth 0 2024-03-25T18:33:37.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.910+0000 7f55f3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:33:37.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.910+0000 7f55f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:37.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.910+0000 7f560171c640 5 monclient: authenticate success, global_id 4571 2024-03-25T18:33:37.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.910+0000 7f560171c640 10 monclient: _renew_subs 2024-03-25T18:33:37.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.910+0000 7f560171c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:33:37.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.910+0000 7f560171c640 1 -- 172.21.15.3:0/2686207084 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55fc1db330 con 0x7f55fc0b4670 2024-03-25T18:33:37.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.910+0000 7f560171c640 10 client.4571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:37.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.910+0000 7f560171c640 10 monclient: _renew_subs 2024-03-25T18:33:37.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.910+0000 7f560171c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:33:37.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.910+0000 7f560171c640 1 -- 172.21.15.3:0/2686207084 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f55fc1e3950 con 0x7f55fc0b4670 2024-03-25T18:33:37.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.910+0000 7f560171c640 1 librados: init done 2024-03-25T18:33:37.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.911+0000 7f560171c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:37.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.911+0000 7f560171c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:37.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.911+0000 7f560171c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:33:37.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:37.911+0000 7f560171c640 1 -- 172.21.15.3:0/2686207084 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55fc1457b0 con 0x7f55fc0b4670 2024-03-25T18:33:38.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.256+0000 7f55f3fff640 1 -- 172.21.15.3:0/2686207084 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55ec001c40 con 0x7f55fc0b4670 2024-03-25T18:33:38.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.256+0000 7f55f3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:38.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.256+0000 7f55f3fff640 1 -- 172.21.15.3:0/2686207084 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f55ec001ac0 con 0x7f55fc0b4670 2024-03-25T18:33:38.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.256+0000 7f55f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:38.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.256+0000 7f55f3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:38.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.256+0000 7f55f3fff640 10 monclient: dump: 2024-03-25T18:33:38.257 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:38.258 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:38.258 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:38.258 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:38.258 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:38.258 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:38.258 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:38.258 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:38.258 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:38.258 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:38.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.256+0000 7f55f9ffb640 4 set_mon_vals no callback set 2024-03-25T18:33:38.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.256+0000 7f55f3fff640 1 -- 172.21.15.3:0/2686207084 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f55ec003b00 con 0x7f55fc0b4670 2024-03-25T18:33:38.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.257+0000 7f55f27fc640 10 monclient: get_auth_request con 0x7f55dc06ddc0 auth_method 0 2024-03-25T18:33:38.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.257+0000 7f55f3fff640 10 client.4571.objecter ms_handle_connect 0x7f55dc06ddc0 2024-03-25T18:33:38.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.263+0000 7f55f3fff640 1 -- 172.21.15.3:0/2686207084 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f55ec054830 con 0x7f55fc0b4670 2024-03-25T18:33:38.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.263+0000 7f55f3fff640 10 client.4571.objecter ms_dispatch 0x7f55fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:38.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.263+0000 7f55f3fff640 3 client.4571.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:38.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.263+0000 7f55f3fff640 3 client.4571.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:38.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.263+0000 7f55f3fff640 20 client.4571.objecter dump_active .. 0 homeless 2024-03-25T18:33:38.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.263+0000 7f55f3fff640 1 -- 172.21.15.3:0/2686207084 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f55ec050340 con 0x7f55fc0b4670 2024-03-25T18:33:38.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.264+0000 7f55f3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:38.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.264+0000 7f55f3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:38.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.517+0000 7f560171c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:38.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.517+0000 7f560171c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:38.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.517+0000 7f560171c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:33:38.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.517+0000 7f560171c640 1 -- 172.21.15.3:0/2686207084 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f55fc0688c0 con 0x7f55fc0b4670 2024-03-25T18:33:38.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.557+0000 7f55f3fff640 1 -- 172.21.15.3:0/2686207084 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f55ec052270 con 0x7f55fc0b4670 2024-03-25T18:33:38.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.557+0000 7f55f3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:38.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.557+0000 7f55f3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:38.559 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:33:38.559 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:38.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.560+0000 7f560171c640 10 librados: watch_flush enter 2024-03-25T18:33:38.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.560+0000 7f560171c640 10 librados: watch_flush exit 2024-03-25T18:33:38.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.560+0000 7f560171c640 20 client.4571.objecter shutdown clearing up homeless session... 2024-03-25T18:33:38.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.560+0000 7f560171c640 10 client.4571.objecter successfully canceled tick 2024-03-25T18:33:38.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.560+0000 7f560171c640 1 -- 172.21.15.3:0/2686207084 >> v1:172.21.15.3:6800/4195140966 conn(0x7f55dc06ddc0 legacy=0x7f55dc070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:38.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.560+0000 7f560171c640 10 monclient: shutdown 2024-03-25T18:33:38.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.560+0000 7f560171c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:38.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.560+0000 7f560171c640 1 -- 172.21.15.3:0/2686207084 >> v1:172.21.15.3:6790/0 conn(0x7f55fc0b4670 legacy=0x7f55fc1450a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:38.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.560+0000 7f560171c640 1 -- 172.21.15.3:0/2686207084 shutdown_connections 2024-03-25T18:33:38.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.560+0000 7f560171c640 1 -- 172.21.15.3:0/2686207084 >> 172.21.15.3:0/2686207084 conn(0x7f55fc0617a0 msgr2=0x7f55fc0b8690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:38.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.561+0000 7f560171c640 1 -- 172.21.15.3:0/2686207084 shutdown_connections 2024-03-25T18:33:38.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.561+0000 7f560171c640 1 -- 172.21.15.3:0/2686207084 wait complete. 2024-03-25T18:33:38.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:38.561+0000 7f560171c640 1 librados: shutdown 2024-03-25T18:33:38.573 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:43.016 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:08:56.745 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:43.021 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:08:56.750 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:43.062 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:08:56.791 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:43.071 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:08:56.800 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:43.076 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:08:56.769 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:43.080 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:08:56.809 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:43.081 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:08:56.773 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:43.103 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:08:56.795 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:43.116 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:08:56.845 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:45.380 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:45.500 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.499+0000 7f2b7973b640 10 monclient: get_monmap_and_config 2024-03-25T18:33:45.500 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.499+0000 7f2b7973b640 10 monclient: build_initial_monmap 2024-03-25T18:33:45.500 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.499+0000 7f2b7973b640 10 monclient: monmap: 2024-03-25T18:33:45.501 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:33:45.501 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:45.501 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:33:45.500416+0000 2024-03-25T18:33:45.501 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:33:45.500416+0000 2024-03-25T18:33:45.501 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:45.501 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:45.501 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:33:45.501 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:33:45.501 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:33:45.501 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:45.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.500+0000 7f2b7973b640 1 Processor -- start 2024-03-25T18:33:45.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.500+0000 7f2b7973b640 1 -- start start 2024-03-25T18:33:45.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.500+0000 7f2b7973b640 10 monclient: init 2024-03-25T18:33:45.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.501+0000 7f2b7973b640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:45.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.501+0000 7f2b7973b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:33:45.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.501+0000 7f2b7973b640 10 monclient(hunting): picked mon.noname-c con 0x7f2b740b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:33:45.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.501+0000 7f2b7973b640 10 monclient(hunting): picked mon.noname-b con 0x7f2b740b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:33:45.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.501+0000 7f2b7973b640 10 monclient(hunting): picked mon.noname-a con 0x7f2b7413c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:33:45.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.501+0000 7f2b7973b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b740ab0e0 con 0x7f2b7413c480 2024-03-25T18:33:45.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.501+0000 7f2b7973b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b740ab3c0 con 0x7f2b740b4ff0 2024-03-25T18:33:45.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.501+0000 7f2b7973b640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b740a9f30 con 0x7f2b740b4670 2024-03-25T18:33:45.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.501+0000 7f2b7973b640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:45.504 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.501+0000 7f2b7973b640 10 monclient(hunting): authenticate will time out at 7879.761740s 2024-03-25T18:33:45.510 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.509+0000 7f2b727fc640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f2b740b4670 0x7f2b741425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:50714/0 (socket says 172.21.15.3:50714) 2024-03-25T18:33:45.510 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.509+0000 7f2b727fc640 1 -- 172.21.15.3:0/1656737733 learned_addr learned my addr 172.21.15.3:0/1656737733 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:33:45.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.524+0000 7f2b727fc640 1 -- 172.21.15.3:0/1656737733 >> v1:172.21.15.3:6790/0 conn(0x7f2b740b4670 legacy=0x7f2b741425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-25T18:33:45.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.524+0000 7f2b727fc640 1 -- 172.21.15.3:0/1656737733 >> v1:172.21.15.3:6790/0 conn(0x7f2b740b4670 legacy=0x7f2b741425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:33:45.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.524+0000 7f2b727fc640 1 --1- 172.21.15.3:0/1656737733 >> v1:172.21.15.3:6790/0 conn(0x7f2b740b4670 0x7f2b741425c0 :-1 s=OPENED pgs=61714 cs=1 l=1).handle_message read tag failed 2024-03-25T18:33:45.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.524+0000 7f2b727fc640 1 --1- 172.21.15.3:0/1656737733 >> v1:172.21.15.3:6790/0 conn(0x7f2b740b4670 0x7f2b741425c0 :-1 s=OPENED pgs=61714 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:33:45.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.524+0000 7f2b71ffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.3:6790/0 2024-03-25T18:33:45.526 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.525+0000 7f2b71ffb640 1 -- 172.21.15.3:0/1656737733 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846476839 0 0) 0x7f2b740ab0e0 con 0x7f2b7413c480 2024-03-25T18:33:45.526 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.525+0000 7f2b71ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:33:45.526 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.525+0000 7f2b71ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:45.526 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.525+0000 7f2b71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:45.526 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.525+0000 7f2b71ffb640 10 monclient(hunting): my global_id is 4620 2024-03-25T18:33:45.526 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.525+0000 7f2b71ffb640 1 -- 172.21.15.3:0/1656737733 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b5c003520 con 0x7f2b7413c480 2024-03-25T18:33:45.527 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.526+0000 7f2b71ffb640 1 -- 172.21.15.3:0/1656737733 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4148799303 0 0) 0x7f2b740ab3c0 con 0x7f2b740b4ff0 2024-03-25T18:33:45.527 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.526+0000 7f2b71ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:33:45.527 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.526+0000 7f2b71ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:45.527 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.526+0000 7f2b71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:45.527 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.526+0000 7f2b71ffb640 10 monclient(hunting): my global_id is 4570 2024-03-25T18:33:45.527 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.526+0000 7f2b71ffb640 1 -- 172.21.15.3:0/1656737733 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b740ab0e0 con 0x7f2b740b4ff0 2024-03-25T18:33:45.544 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.543+0000 7f2b71ffb640 1 -- 172.21.15.3:0/1656737733 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1757936074 0 0) 0x7f2b740ab0e0 con 0x7f2b740b4ff0 2024-03-25T18:33:45.544 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.543+0000 7f2b71ffb640 1 -- 172.21.15.3:0/1656737733 >> v1:172.21.15.3:6790/0 conn(0x7f2b740b4670 legacy=0x7f2b741425c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:33:45.545 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.543+0000 7f2b71ffb640 1 -- 172.21.15.3:0/1656737733 >> v1:172.21.15.3:6789/0 conn(0x7f2b7413c480 legacy=0x7f2b7414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:45.545 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.543+0000 7f2b71ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:33:45.545 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.543+0000 7f2b71ffb640 1 monclient: found mon.noname-c 2024-03-25T18:33:45.545 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.543+0000 7f2b71ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:45.545 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.543+0000 7f2b71ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:33:45.545 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.543+0000 7f2b71ffb640 1 -- 172.21.15.3:0/1656737733 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b740a9b70 con 0x7f2b740b4ff0 2024-03-25T18:33:45.545 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.543+0000 7f2b71ffb640 10 monclient: _finish_auth 0 2024-03-25T18:33:45.545 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.543+0000 7f2b71ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:33:45.545 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.543+0000 7f2b71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:45.545 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.543+0000 7f2b71ffb640 1 -- 172.21.15.3:0/1656737733 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f2b64001630 con 0x7f2b740b4ff0 2024-03-25T18:33:45.545 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.543+0000 7f2b71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:45.545 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.543+0000 7f2b71ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:45.545 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.543+0000 7f2b71ffb640 10 monclient: dump: 2024-03-25T18:33:45.546 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:45.546 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:45.546 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:45.546 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:45.546 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:45.546 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:45.546 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:45.546 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:45.546 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:45.546 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:45.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.543+0000 7f2b7973b640 5 monclient: authenticate success, global_id 4570 2024-03-25T18:33:45.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.543+0000 7f2b7973b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:45.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.566+0000 7f2b71ffb640 1 -- 172.21.15.3:0/1656737733 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b64001790 con 0x7f2b740b4ff0 2024-03-25T18:33:45.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.566+0000 7f2b71ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:45.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.566+0000 7f2b71ffb640 1 -- 172.21.15.3:0/1656737733 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f2b64001ba0 con 0x7f2b740b4ff0 2024-03-25T18:33:45.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.566+0000 7f2b7973b640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:45.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.566+0000 7f2b7973b640 4 set_mon_vals no callback set 2024-03-25T18:33:45.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.566+0000 7f2b71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:45.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.566+0000 7f2b71ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:45.568 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.566+0000 7f2b71ffb640 10 monclient: dump: 2024-03-25T18:33:45.568 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:45.568 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:45.568 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:45.568 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:45.568 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:45.568 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:45.568 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:45.568 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:45.568 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:45.568 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:45.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.566+0000 7f2b7973b640 10 monclient: shutdown 2024-03-25T18:33:45.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.566+0000 7f2b7973b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:45.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.566+0000 7f2b7973b640 1 -- 172.21.15.3:0/1656737733 >> v1:172.21.15.16:6789/0 conn(0x7f2b740b4ff0 legacy=0x7f2b7413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:45.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.566+0000 7f2b7973b640 1 -- 172.21.15.3:0/1656737733 shutdown_connections 2024-03-25T18:33:45.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.566+0000 7f2b7973b640 1 -- 172.21.15.3:0/1656737733 >> 172.21.15.3:0/1656737733 conn(0x7f2b740617a0 msgr2=0x7f2b74061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:45.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.566+0000 7f2b7973b640 1 -- 172.21.15.3:0/1656737733 shutdown_connections 2024-03-25T18:33:45.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.567+0000 7f2b7973b640 1 -- 172.21.15.3:0/1656737733 wait complete. 2024-03-25T18:33:45.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.567+0000 7f2b7973b640 10 monclient: build_initial_monmap 2024-03-25T18:33:45.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.567+0000 7f2b7973b640 10 monclient: monmap: 2024-03-25T18:33:45.571 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:33:45.571 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:45.571 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:33:45.571 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:33:45.571 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:45.571 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:33:45.571 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:33:45.571 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:33:45.571 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:33:45.571 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:45.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.568+0000 7f2b7973b640 1 librados: starting msgr at 2024-03-25T18:33:45.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.568+0000 7f2b7973b640 1 librados: starting objecter 2024-03-25T18:33:45.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.568+0000 7f2b7973b640 1 Processor -- start 2024-03-25T18:33:45.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.568+0000 7f2b7973b640 1 -- start start 2024-03-25T18:33:45.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.568+0000 7f2b7973b640 1 librados: setting wanted keys 2024-03-25T18:33:45.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.568+0000 7f2b7973b640 1 librados: calling monclient init 2024-03-25T18:33:45.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.568+0000 7f2b7973b640 10 monclient: init 2024-03-25T18:33:45.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.569+0000 7f2b7973b640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:45.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.569+0000 7f2b7973b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:33:45.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.569+0000 7f2b7973b640 10 monclient(hunting): picked mon.noname-c con 0x7f2b740b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:33:45.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.569+0000 7f2b7973b640 10 monclient(hunting): picked mon.noname-b con 0x7f2b740b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:33:45.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.569+0000 7f2b7973b640 10 monclient(hunting): picked mon.noname-a con 0x7f2b7413c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:33:45.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.569+0000 7f2b7973b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b74130e20 con 0x7f2b7413c480 2024-03-25T18:33:45.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.569+0000 7f2b7973b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b74137c30 con 0x7f2b740b4ff0 2024-03-25T18:33:45.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.569+0000 7f2b7973b640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b74145620 con 0x7f2b740b4670 2024-03-25T18:33:45.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.569+0000 7f2b7973b640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:45.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.569+0000 7f2b7973b640 10 monclient(hunting): authenticate will time out at 7879.829870s 2024-03-25T18:33:45.585 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.584+0000 7f2b737fe640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f2b7413c480 0x7f2b74130710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:36596/0 (socket says 172.21.15.3:36596) 2024-03-25T18:33:45.586 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.584+0000 7f2b72ffd640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f2b740b4670 0x7f2b74131110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:50746/0 (socket says 172.21.15.3:50746) 2024-03-25T18:33:45.586 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.584+0000 7f2b737fe640 1 -- 172.21.15.3:0/3001192931 learned_addr learned my addr 172.21.15.3:0/3001192931 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:33:45.586 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.585+0000 7f2b5b7fe640 10 client.?.objecter ms_handle_connect 0x7f2b740b4670 2024-03-25T18:33:45.586 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.585+0000 7f2b5b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:45.586 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.585+0000 7f2b5b7fe640 10 client.?.objecter ms_handle_connect 0x7f2b7413c480 2024-03-25T18:33:45.586 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.585+0000 7f2b5b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:45.591 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.590+0000 7f2b5b7fe640 10 client.?.objecter ms_handle_connect 0x7f2b740b4ff0 2024-03-25T18:33:45.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.590+0000 7f2b5b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:45.596 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.595+0000 7f2b5b7fe640 1 -- 172.21.15.3:0/3001192931 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1913353400 0 0) 0x7f2b74130e20 con 0x7f2b7413c480 2024-03-25T18:33:45.596 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.595+0000 7f2b5b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:33:45.596 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.595+0000 7f2b5b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:45.596 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.595+0000 7f2b5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:45.596 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.595+0000 7f2b5b7fe640 10 monclient(hunting): my global_id is 4623 2024-03-25T18:33:45.596 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.595+0000 7f2b5b7fe640 1 -- 172.21.15.3:0/3001192931 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b60004eb0 con 0x7f2b7413c480 2024-03-25T18:33:45.597 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.595+0000 7f2b5b7fe640 1 -- 172.21.15.3:0/3001192931 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3608978349 0 0) 0x7f2b74145620 con 0x7f2b740b4670 2024-03-25T18:33:45.597 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.595+0000 7f2b5b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:33:45.597 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.595+0000 7f2b5b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:45.597 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.595+0000 7f2b5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:45.597 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.595+0000 7f2b5b7fe640 10 monclient(hunting): my global_id is 4577 2024-03-25T18:33:45.597 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.595+0000 7f2b5b7fe640 1 -- 172.21.15.3:0/3001192931 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b74130e20 con 0x7f2b740b4670 2024-03-25T18:33:45.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.599+0000 7f2b5b7fe640 1 -- 172.21.15.3:0/3001192931 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2776481222 0 0) 0x7f2b74137c30 con 0x7f2b740b4ff0 2024-03-25T18:33:45.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.599+0000 7f2b5b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:33:45.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.599+0000 7f2b5b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:45.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.599+0000 7f2b5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:45.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.599+0000 7f2b5b7fe640 10 monclient(hunting): my global_id is 4573 2024-03-25T18:33:45.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.599+0000 7f2b5b7fe640 1 -- 172.21.15.3:0/3001192931 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b74145620 con 0x7f2b740b4ff0 2024-03-25T18:33:45.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.611+0000 7f2b5b7fe640 1 -- 172.21.15.3:0/3001192931 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2681869674 0 0) 0x7f2b60004eb0 con 0x7f2b7413c480 2024-03-25T18:33:45.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.611+0000 7f2b5b7fe640 1 -- 172.21.15.3:0/3001192931 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b74137c30 con 0x7f2b7413c480 2024-03-25T18:33:45.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.611+0000 7f2b5b7fe640 1 -- 172.21.15.3:0/3001192931 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f2b6c001c30 con 0x7f2b7413c480 2024-03-25T18:33:45.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.611+0000 7f2b5b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:45.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.611+0000 7f2b5b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:45.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.611+0000 7f2b5b7fe640 10 monclient(hunting): dump: 2024-03-25T18:33:45.613 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:45.613 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:45.613 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:45.613 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:45.613 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:45.613 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:45.613 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:45.613 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:45.614 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:45.614 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:45.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.611+0000 7f2b5b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:45.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.613+0000 7f2b5b7fe640 1 -- 172.21.15.3:0/3001192931 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1342681047 0 0) 0x7f2b74145620 con 0x7f2b740b4ff0 2024-03-25T18:33:45.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.613+0000 7f2b5b7fe640 1 -- 172.21.15.3:0/3001192931 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b60004eb0 con 0x7f2b740b4ff0 2024-03-25T18:33:45.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.613+0000 7f2b5b7fe640 1 -- 172.21.15.3:0/3001192931 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f2b68001830 con 0x7f2b740b4ff0 2024-03-25T18:33:45.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.613+0000 7f2b5b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:45.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.613+0000 7f2b5b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:45.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.613+0000 7f2b5b7fe640 10 monclient(hunting): dump: 2024-03-25T18:33:45.615 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:45.615 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:45.615 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:45.615 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:45.615 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:45.615 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:45.615 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:45.616 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:45.616 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:45.616 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:45.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.623+0000 7f2b5b7fe640 1 -- 172.21.15.3:0/3001192931 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 977711564 0 0) 0x7f2b60004eb0 con 0x7f2b740b4ff0 2024-03-25T18:33:45.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.623+0000 7f2b5b7fe640 1 -- 172.21.15.3:0/3001192931 >> v1:172.21.15.3:6790/0 conn(0x7f2b740b4670 legacy=0x7f2b74131110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:45.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.623+0000 7f2b5b7fe640 1 -- 172.21.15.3:0/3001192931 >> v1:172.21.15.3:6789/0 conn(0x7f2b7413c480 legacy=0x7f2b74130710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:45.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.623+0000 7f2b5b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:33:45.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.623+0000 7f2b5b7fe640 1 monclient: found mon.b 2024-03-25T18:33:45.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.623+0000 7f2b5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:45.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.623+0000 7f2b5b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:33:45.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.623+0000 7f2b5b7fe640 1 -- 172.21.15.3:0/3001192931 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b74145190 con 0x7f2b740b4ff0 2024-03-25T18:33:45.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.623+0000 7f2b5b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:33:45.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.623+0000 7f2b5b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:33:45.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.623+0000 7f2b5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:45.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.623+0000 7f2b7973b640 5 monclient: authenticate success, global_id 4573 2024-03-25T18:33:45.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.623+0000 7f2b7973b640 10 monclient: _renew_subs 2024-03-25T18:33:45.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.623+0000 7f2b7973b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:33:45.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.623+0000 7f2b7973b640 1 -- 172.21.15.3:0/3001192931 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b74145380 con 0x7f2b740b4ff0 2024-03-25T18:33:45.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.623+0000 7f2b7973b640 10 client.4573.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:45.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.623+0000 7f2b7973b640 10 monclient: _renew_subs 2024-03-25T18:33:45.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.623+0000 7f2b7973b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:33:45.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.623+0000 7f2b7973b640 1 -- 172.21.15.3:0/3001192931 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b74131a00 con 0x7f2b740b4ff0 2024-03-25T18:33:45.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.623+0000 7f2b7973b640 1 librados: init done 2024-03-25T18:33:45.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.625+0000 7f2b7973b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:45.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.625+0000 7f2b7973b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:45.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.625+0000 7f2b7973b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:33:45.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.625+0000 7f2b7973b640 1 -- 172.21.15.3:0/3001192931 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b7406dbd0 con 0x7f2b740b4ff0 2024-03-25T18:33:45.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.922+0000 7f2b5b7fe640 1 -- 172.21.15.3:0/3001192931 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b68001c80 con 0x7f2b740b4ff0 2024-03-25T18:33:45.924 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.922+0000 7f2b5b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:45.924 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.922+0000 7f2b5b7fe640 1 -- 172.21.15.3:0/3001192931 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f2b68004980 con 0x7f2b740b4ff0 2024-03-25T18:33:45.924 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.922+0000 7f2b5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:45.924 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.922+0000 7f2b5b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:45.924 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.922+0000 7f2b5b7fe640 10 monclient: dump: 2024-03-25T18:33:45.924 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:45.924 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:45.924 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:45.924 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:45.924 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:45.924 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:45.924 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:45.924 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:45.925 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:45.925 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:45.925 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.922+0000 7f2b717fa640 4 set_mon_vals no callback set 2024-03-25T18:33:45.925 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.923+0000 7f2b5b7fe640 1 -- 172.21.15.3:0/3001192931 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f2b68004ba0 con 0x7f2b740b4ff0 2024-03-25T18:33:45.925 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.923+0000 7f2b72ffd640 10 monclient: get_auth_request con 0x7f2b6006df70 auth_method 0 2024-03-25T18:33:45.926 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.924+0000 7f2b5b7fe640 10 client.4573.objecter ms_handle_connect 0x7f2b6006df70 2024-03-25T18:33:45.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.929+0000 7f2b5b7fe640 1 -- 172.21.15.3:0/3001192931 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f2b680546f0 con 0x7f2b740b4ff0 2024-03-25T18:33:45.931 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.929+0000 7f2b5b7fe640 10 client.4573.objecter ms_dispatch 0x7f2b74000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:45.931 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.929+0000 7f2b5b7fe640 3 client.4573.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:45.931 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.929+0000 7f2b5b7fe640 3 client.4573.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:45.931 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.929+0000 7f2b5b7fe640 20 client.4573.objecter dump_active .. 0 homeless 2024-03-25T18:33:45.931 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.930+0000 7f2b5b7fe640 1 -- 172.21.15.3:0/3001192931 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2b68050200 con 0x7f2b740b4ff0 2024-03-25T18:33:45.931 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.930+0000 7f2b5b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:45.931 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:45.930+0000 7f2b5b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:46.148 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:46.146+0000 7f2b7973b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:46.148 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:46.146+0000 7f2b7973b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:46.148 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:46.146+0000 7f2b7973b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:33:46.148 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:46.146+0000 7f2b7973b640 1 -- 172.21.15.3:0/3001192931 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b74146790 con 0x7f2b740b4ff0 2024-03-25T18:33:46.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:46.210+0000 7f2b5b7fe640 1 -- 172.21.15.3:0/3001192931 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f2b68052130 con 0x7f2b740b4ff0 2024-03-25T18:33:46.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:46.210+0000 7f2b5b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:46.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:46.210+0000 7f2b5b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:46.211 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:33:46.211 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:46.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:46.212+0000 7f2b7973b640 10 librados: watch_flush enter 2024-03-25T18:33:46.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:46.212+0000 7f2b7973b640 10 librados: watch_flush exit 2024-03-25T18:33:46.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:46.212+0000 7f2b7973b640 20 client.4573.objecter shutdown clearing up homeless session... 2024-03-25T18:33:46.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:46.212+0000 7f2b7973b640 10 client.4573.objecter successfully canceled tick 2024-03-25T18:33:46.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:46.212+0000 7f2b7973b640 1 -- 172.21.15.3:0/3001192931 >> v1:172.21.15.3:6800/4195140966 conn(0x7f2b6006df70 legacy=0x7f2b60070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:46.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:46.212+0000 7f2b7973b640 10 monclient: shutdown 2024-03-25T18:33:46.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:46.212+0000 7f2b7973b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:46.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:46.212+0000 7f2b7973b640 1 -- 172.21.15.3:0/3001192931 >> v1:172.21.15.16:6789/0 conn(0x7f2b740b4ff0 legacy=0x7f2b74130000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:46.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:46.213+0000 7f2b7973b640 1 -- 172.21.15.3:0/3001192931 shutdown_connections 2024-03-25T18:33:46.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:46.213+0000 7f2b7973b640 1 -- 172.21.15.3:0/3001192931 >> 172.21.15.3:0/3001192931 conn(0x7f2b740617a0 msgr2=0x7f2b74140670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:46.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:46.213+0000 7f2b7973b640 1 -- 172.21.15.3:0/3001192931 shutdown_connections 2024-03-25T18:33:46.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:46.213+0000 7f2b7973b640 1 -- 172.21.15.3:0/3001192931 wait complete. 2024-03-25T18:33:46.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:46.213+0000 7f2b7973b640 1 librados: shutdown 2024-03-25T18:33:46.226 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:53.032 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:53.144 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.143+0000 7f70206e6640 10 monclient: get_monmap_and_config 2024-03-25T18:33:53.144 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.143+0000 7f70206e6640 10 monclient: build_initial_monmap 2024-03-25T18:33:53.144 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.143+0000 7f70206e6640 10 monclient: monmap: 2024-03-25T18:33:53.144 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:33:53.144 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:53.144 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:33:53.144277+0000 2024-03-25T18:33:53.144 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:33:53.144277+0000 2024-03-25T18:33:53.144 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:53.144 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:53.144 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:33:53.145 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:33:53.145 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:33:53.145 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:53.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.143+0000 7f70206e6640 1 Processor -- start 2024-03-25T18:33:53.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.143+0000 7f70206e6640 1 -- start start 2024-03-25T18:33:53.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.143+0000 7f70206e6640 10 monclient: init 2024-03-25T18:33:53.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.144+0000 7f70206e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:53.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.144+0000 7f70206e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:33:53.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.144+0000 7f70206e6640 10 monclient(hunting): picked mon.noname-b con 0x7f701806c3e0 addr v1:172.21.15.3:6790/0 2024-03-25T18:33:53.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.144+0000 7f70206e6640 10 monclient(hunting): picked mon.noname-a con 0x7f70180b0e00 addr v1:172.21.15.3:6789/0 2024-03-25T18:33:53.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.144+0000 7f70206e6640 10 monclient(hunting): picked mon.noname-c con 0x7f70181465b0 addr v1:172.21.15.16:6789/0 2024-03-25T18:33:53.147 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.144+0000 7f70206e6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70181469d0 con 0x7f70180b0e00 2024-03-25T18:33:53.147 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.145+0000 7f70206e6640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70180adfc0 con 0x7f70181465b0 2024-03-25T18:33:53.147 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.145+0000 7f70206e6640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70180ae2a0 con 0x7f701806c3e0 2024-03-25T18:33:53.147 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.145+0000 7f70206e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:53.147 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.145+0000 7f70206e6640 10 monclient(hunting): authenticate will time out at 7887.405375s 2024-03-25T18:33:53.159 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.158+0000 7f701ec83640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f70181465b0 0x7f701814ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:54080/0 (socket says 172.21.15.3:54080) 2024-03-25T18:33:53.159 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.158+0000 7f701ec83640 1 -- 172.21.15.3:0/3245670701 learned_addr learned my addr 172.21.15.3:0/3245670701 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:33:53.172 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.171+0000 7f701d480640 1 -- 172.21.15.3:0/3245670701 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 788694161 0 0) 0x7f70180adfc0 con 0x7f70181465b0 2024-03-25T18:33:53.172 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.171+0000 7f701d480640 10 monclient(hunting): _negotiate 2024-03-25T18:33:53.172 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.171+0000 7f701d480640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:53.172 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.171+0000 7f701d480640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:53.172 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.171+0000 7f701d480640 10 monclient(hunting): my global_id is 4576 2024-03-25T18:33:53.172 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.171+0000 7f701d480640 1 -- 172.21.15.3:0/3245670701 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7004003520 con 0x7f70181465b0 2024-03-25T18:33:53.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.188+0000 7f701d480640 1 -- 172.21.15.3:0/3245670701 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1144162880 0 0) 0x7f70181469d0 con 0x7f70180b0e00 2024-03-25T18:33:53.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.188+0000 7f701d480640 10 monclient(hunting): _negotiate 2024-03-25T18:33:53.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.188+0000 7f701d480640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:53.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.188+0000 7f701d480640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:53.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.188+0000 7f701d480640 10 monclient(hunting): my global_id is 4626 2024-03-25T18:33:53.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.188+0000 7f701d480640 1 -- 172.21.15.3:0/3245670701 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70180adfc0 con 0x7f70180b0e00 2024-03-25T18:33:53.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.189+0000 7f701d480640 1 -- 172.21.15.3:0/3245670701 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1448207966 0 0) 0x7f7004003520 con 0x7f70181465b0 2024-03-25T18:33:53.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.189+0000 7f701d480640 1 -- 172.21.15.3:0/3245670701 >> v1:172.21.15.3:6790/0 conn(0x7f701806c3e0 legacy=0x7f70180b06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:53.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.189+0000 7f701d480640 1 -- 172.21.15.3:0/3245670701 >> v1:172.21.15.3:6789/0 conn(0x7f70180b0e00 legacy=0x7f7018145ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:53.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.189+0000 7f701d480640 10 monclient: _finish_hunting 0 2024-03-25T18:33:53.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.189+0000 7f701d480640 1 monclient: found mon.noname-c 2024-03-25T18:33:53.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.189+0000 7f701d480640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:53.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.189+0000 7f701d480640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:33:53.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.189+0000 7f701d480640 1 -- 172.21.15.3:0/3245670701 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f701806d210 con 0x7f70181465b0 2024-03-25T18:33:53.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.189+0000 7f701d480640 10 monclient: _finish_auth 0 2024-03-25T18:33:53.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.189+0000 7f701d480640 10 monclient: _check_auth_tickets 2024-03-25T18:33:53.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.189+0000 7f701d480640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:53.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.189+0000 7f701d480640 1 -- 172.21.15.3:0/3245670701 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7010001a10 con 0x7f70181465b0 2024-03-25T18:33:53.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.189+0000 7f70206e6640 5 monclient: authenticate success, global_id 4576 2024-03-25T18:33:53.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.189+0000 7f70206e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:53.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.189+0000 7f701d480640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:53.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.189+0000 7f701d480640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:53.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.189+0000 7f701d480640 10 monclient: dump: 2024-03-25T18:33:53.192 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:53.192 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:53.192 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:53.192 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:53.192 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:53.192 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:53.192 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:53.193 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:53.193 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:53.193 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:53.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.189+0000 7f70206e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:53.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.198+0000 7f701d480640 1 -- 172.21.15.3:0/3245670701 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7010001b70 con 0x7f70181465b0 2024-03-25T18:33:53.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.198+0000 7f701d480640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:53.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.198+0000 7f701d480640 1 -- 172.21.15.3:0/3245670701 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7010001f00 con 0x7f70181465b0 2024-03-25T18:33:53.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.198+0000 7f70206e6640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:53.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.198+0000 7f70206e6640 4 set_mon_vals no callback set 2024-03-25T18:33:53.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.198+0000 7f701d480640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:53.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.198+0000 7f701d480640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:53.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.198+0000 7f701d480640 10 monclient: dump: 2024-03-25T18:33:53.200 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:53.200 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:53.200 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:53.200 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:53.201 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:53.201 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:53.201 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:53.201 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:53.201 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:53.201 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:53.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.199+0000 7f70206e6640 10 monclient: shutdown 2024-03-25T18:33:53.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.199+0000 7f70206e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:53.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.199+0000 7f70206e6640 1 -- 172.21.15.3:0/3245670701 >> v1:172.21.15.16:6789/0 conn(0x7f70181465b0 legacy=0x7f701814ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:53.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.199+0000 7f70206e6640 1 -- 172.21.15.3:0/3245670701 shutdown_connections 2024-03-25T18:33:53.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.199+0000 7f70206e6640 1 -- 172.21.15.3:0/3245670701 >> 172.21.15.3:0/3245670701 conn(0x7f70180b5cf0 msgr2=0x7f70180b6100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:53.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.199+0000 7f70206e6640 1 -- 172.21.15.3:0/3245670701 shutdown_connections 2024-03-25T18:33:53.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.199+0000 7f70206e6640 1 -- 172.21.15.3:0/3245670701 wait complete. 2024-03-25T18:33:53.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.200+0000 7f70206e6640 10 monclient: build_initial_monmap 2024-03-25T18:33:53.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.200+0000 7f70206e6640 10 monclient: monmap: 2024-03-25T18:33:53.203 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:33:53.204 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:53.204 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:33:53.204 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:33:53.204 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:53.204 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:33:53.204 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:33:53.204 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:33:53.204 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:33:53.204 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:53.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.200+0000 7f70206e6640 1 librados: starting msgr at 2024-03-25T18:33:53.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.200+0000 7f70206e6640 1 librados: starting objecter 2024-03-25T18:33:53.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.200+0000 7f70206e6640 1 Processor -- start 2024-03-25T18:33:53.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.200+0000 7f70206e6640 1 -- start start 2024-03-25T18:33:53.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.200+0000 7f70206e6640 1 librados: setting wanted keys 2024-03-25T18:33:53.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.200+0000 7f70206e6640 1 librados: calling monclient init 2024-03-25T18:33:53.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.200+0000 7f70206e6640 10 monclient: init 2024-03-25T18:33:53.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.201+0000 7f70206e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:53.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.201+0000 7f70206e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:33:53.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.201+0000 7f70206e6640 10 monclient(hunting): picked mon.noname-a con 0x7f701806c3e0 addr v1:172.21.15.3:6789/0 2024-03-25T18:33:53.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.201+0000 7f70206e6640 10 monclient(hunting): picked mon.noname-c con 0x7f70180b0e00 addr v1:172.21.15.3:6790/0 2024-03-25T18:33:53.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.201+0000 7f70206e6640 10 monclient(hunting): picked mon.noname-b con 0x7f70181465b0 addr v1:172.21.15.16:6789/0 2024-03-25T18:33:53.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.201+0000 7f70206e6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7018141940 con 0x7f701806c3e0 2024-03-25T18:33:53.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.201+0000 7f70206e6640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7018141690 con 0x7f70181465b0 2024-03-25T18:33:53.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.201+0000 7f70206e6640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7018121ac0 con 0x7f70180b0e00 2024-03-25T18:33:53.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.201+0000 7f70206e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:53.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.201+0000 7f70206e6640 10 monclient(hunting): authenticate will time out at 7887.462279s 2024-03-25T18:33:53.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.221+0000 7f701e482640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f701806c3e0 0x7f70181d72f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:35228/0 (socket says 172.21.15.3:35228) 2024-03-25T18:33:53.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.221+0000 7f701e482640 1 -- 172.21.15.3:0/1028316314 learned_addr learned my addr 172.21.15.3:0/1028316314 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:33:53.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.221+0000 7f700effd640 10 client.?.objecter ms_handle_connect 0x7f701806c3e0 2024-03-25T18:33:53.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.221+0000 7f700effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:53.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.223+0000 7f700effd640 10 client.?.objecter ms_handle_connect 0x7f70180b0e00 2024-03-25T18:33:53.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.223+0000 7f700effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:53.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.226+0000 7f700effd640 10 client.?.objecter ms_handle_connect 0x7f70181465b0 2024-03-25T18:33:53.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.226+0000 7f700effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:53.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.237+0000 7f700effd640 1 -- 172.21.15.3:0/1028316314 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3807226272 0 0) 0x7f7018141690 con 0x7f70181465b0 2024-03-25T18:33:53.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.237+0000 7f700effd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:53.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.237+0000 7f700effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:53.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.237+0000 7f700effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:53.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.237+0000 7f700effd640 10 monclient(hunting): my global_id is 4579 2024-03-25T18:33:53.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.237+0000 7f700effd640 1 -- 172.21.15.3:0/1028316314 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ffc004eb0 con 0x7f70181465b0 2024-03-25T18:33:53.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.238+0000 7f700effd640 1 -- 172.21.15.3:0/1028316314 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3176187482 0 0) 0x7f7018121ac0 con 0x7f70180b0e00 2024-03-25T18:33:53.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.238+0000 7f700effd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:53.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.238+0000 7f700effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:53.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.238+0000 7f700effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:53.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.238+0000 7f700effd640 10 monclient(hunting): my global_id is 4583 2024-03-25T18:33:53.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.238+0000 7f700effd640 1 -- 172.21.15.3:0/1028316314 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7018141690 con 0x7f70180b0e00 2024-03-25T18:33:53.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.242+0000 7f700effd640 1 -- 172.21.15.3:0/1028316314 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3161306872 0 0) 0x7f7018141940 con 0x7f701806c3e0 2024-03-25T18:33:53.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.242+0000 7f700effd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:53.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.242+0000 7f700effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:53.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.242+0000 7f700effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:53.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.242+0000 7f700effd640 10 monclient(hunting): my global_id is 4629 2024-03-25T18:33:53.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.242+0000 7f700effd640 1 -- 172.21.15.3:0/1028316314 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7018121ac0 con 0x7f701806c3e0 2024-03-25T18:33:53.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.254+0000 7f700effd640 1 -- 172.21.15.3:0/1028316314 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2647635574 0 0) 0x7f6ffc004eb0 con 0x7f70181465b0 2024-03-25T18:33:53.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.254+0000 7f700effd640 1 -- 172.21.15.3:0/1028316314 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7018141940 con 0x7f70181465b0 2024-03-25T18:33:53.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.254+0000 7f700effd640 1 -- 172.21.15.3:0/1028316314 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7010001e80 con 0x7f70181465b0 2024-03-25T18:33:53.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.254+0000 7f700effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:53.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.254+0000 7f700effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:53.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.254+0000 7f700effd640 10 monclient(hunting): dump: 2024-03-25T18:33:53.255 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:53.255 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:53.255 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:53.255 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:53.256 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:53.256 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:53.256 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:53.256 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:53.256 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:53.256 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:53.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.254+0000 7f700effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:53.262 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.261+0000 7f700effd640 1 -- 172.21.15.3:0/1028316314 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3921136957 0 0) 0x7f7018121ac0 con 0x7f701806c3e0 2024-03-25T18:33:53.262 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.261+0000 7f700effd640 1 -- 172.21.15.3:0/1028316314 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ffc004eb0 con 0x7f701806c3e0 2024-03-25T18:33:53.262 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.261+0000 7f700effd640 1 -- 172.21.15.3:0/1028316314 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7014001970 con 0x7f701806c3e0 2024-03-25T18:33:53.262 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.261+0000 7f700effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:53.262 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.261+0000 7f700effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:53.262 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.261+0000 7f700effd640 10 monclient(hunting): dump: 2024-03-25T18:33:53.262 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:53.262 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:53.262 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:53.263 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:53.263 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:53.263 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:53.263 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:53.263 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:53.263 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:53.263 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:53.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.263+0000 7f700effd640 1 -- 172.21.15.3:0/1028316314 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1051491243 0 0) 0x7f7018141690 con 0x7f70180b0e00 2024-03-25T18:33:53.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.263+0000 7f700effd640 1 -- 172.21.15.3:0/1028316314 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7018121ac0 con 0x7f70180b0e00 2024-03-25T18:33:53.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.263+0000 7f700effd640 1 -- 172.21.15.3:0/1028316314 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7008001930 con 0x7f70180b0e00 2024-03-25T18:33:53.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.263+0000 7f700effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:53.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.263+0000 7f700effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:53.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.263+0000 7f700effd640 10 monclient(hunting): dump: 2024-03-25T18:33:53.264 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:53.264 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:53.264 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:53.264 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:53.265 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:53.265 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:53.265 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:53.265 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:53.265 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:53.265 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:53.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.264+0000 7f700effd640 1 -- 172.21.15.3:0/1028316314 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3487177984 0 0) 0x7f7018141940 con 0x7f70181465b0 2024-03-25T18:33:53.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.264+0000 7f700effd640 1 -- 172.21.15.3:0/1028316314 >> v1:172.21.15.3:6790/0 conn(0x7f70180b0e00 legacy=0x7f7018140870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:53.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.264+0000 7f700effd640 1 -- 172.21.15.3:0/1028316314 >> v1:172.21.15.3:6789/0 conn(0x7f701806c3e0 legacy=0x7f70181d72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:53.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.264+0000 7f700effd640 10 monclient: _finish_hunting 0 2024-03-25T18:33:53.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.264+0000 7f700effd640 1 monclient: found mon.b 2024-03-25T18:33:53.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.264+0000 7f700effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:53.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.264+0000 7f700effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:33:53.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.264+0000 7f700effd640 1 -- 172.21.15.3:0/1028316314 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70181d7a00 con 0x7f70181465b0 2024-03-25T18:33:53.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.264+0000 7f700effd640 10 monclient: _finish_auth 0 2024-03-25T18:33:53.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.264+0000 7f700effd640 10 monclient: _check_auth_tickets 2024-03-25T18:33:53.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.264+0000 7f700effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:53.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.264+0000 7f70206e6640 5 monclient: authenticate success, global_id 4579 2024-03-25T18:33:53.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.264+0000 7f70206e6640 10 monclient: _renew_subs 2024-03-25T18:33:53.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.264+0000 7f70206e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:33:53.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.264+0000 7f70206e6640 1 -- 172.21.15.3:0/1028316314 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7018122330 con 0x7f70181465b0 2024-03-25T18:33:53.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.264+0000 7f70206e6640 10 client.4579.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:53.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.264+0000 7f70206e6640 10 monclient: _renew_subs 2024-03-25T18:33:53.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.264+0000 7f70206e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:33:53.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.264+0000 7f70206e6640 1 -- 172.21.15.3:0/1028316314 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7018071090 con 0x7f70181465b0 2024-03-25T18:33:53.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.264+0000 7f70206e6640 1 librados: init done 2024-03-25T18:33:53.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.265+0000 7f70206e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:53.268 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.265+0000 7f70206e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:53.268 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.265+0000 7f70206e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:33:53.268 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.265+0000 7f70206e6640 1 -- 172.21.15.3:0/1028316314 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7018121c50 con 0x7f70181465b0 2024-03-25T18:33:53.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.563+0000 7f700effd640 1 -- 172.21.15.3:0/1028316314 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7010001fe0 con 0x7f70181465b0 2024-03-25T18:33:53.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.563+0000 7f700effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:53.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.563+0000 7f700effd640 1 -- 172.21.15.3:0/1028316314 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f7010004fb0 con 0x7f70181465b0 2024-03-25T18:33:53.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.563+0000 7f700effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:53.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.563+0000 7f700effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:53.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.563+0000 7f700effd640 10 monclient: dump: 2024-03-25T18:33:53.565 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:33:53.565 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:33:53.565 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:53.565 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:33:53.565 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:53.565 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:33:53.565 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:33:53.565 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:33:53.565 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:33:53.565 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:33:53.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.563+0000 7f701cc7f640 4 set_mon_vals no callback set 2024-03-25T18:33:53.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.564+0000 7f700effd640 1 -- 172.21.15.3:0/1028316314 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f701001af80 con 0x7f70181465b0 2024-03-25T18:33:53.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.564+0000 7f701e482640 10 monclient: get_auth_request con 0x7f6ffc06e0b0 auth_method 0 2024-03-25T18:33:53.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.565+0000 7f700effd640 10 client.4579.objecter ms_handle_connect 0x7f6ffc06e0b0 2024-03-25T18:33:53.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.571+0000 7f700effd640 1 -- 172.21.15.3:0/1028316314 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f7010054b10 con 0x7f70181465b0 2024-03-25T18:33:53.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.571+0000 7f700effd640 10 client.4579.objecter ms_dispatch 0x7f7018000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:53.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.571+0000 7f700effd640 3 client.4579.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:53.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.571+0000 7f700effd640 3 client.4579.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:53.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.571+0000 7f700effd640 20 client.4579.objecter dump_active .. 0 homeless 2024-03-25T18:33:53.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.571+0000 7f700effd640 1 -- 172.21.15.3:0/1028316314 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7010050620 con 0x7f70181465b0 2024-03-25T18:33:53.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.571+0000 7f700effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:53.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.571+0000 7f700effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:53.810 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.809+0000 7f70206e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:53.810 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.809+0000 7f70206e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:53.811 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.809+0000 7f70206e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:33:53.811 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.809+0000 7f70206e6640 1 -- 172.21.15.3:0/1028316314 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f701804f730 con 0x7f70181465b0 2024-03-25T18:33:53.902 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.901+0000 7f700effd640 1 -- 172.21.15.3:0/1028316314 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f7010052550 con 0x7f70181465b0 2024-03-25T18:33:53.902 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.901+0000 7f700effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:53.902 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.901+0000 7f700effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:53.902 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:33:53.903 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:53.905 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.903+0000 7f70206e6640 10 librados: watch_flush enter 2024-03-25T18:33:53.905 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.903+0000 7f70206e6640 10 librados: watch_flush exit 2024-03-25T18:33:53.905 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.903+0000 7f70206e6640 20 client.4579.objecter shutdown clearing up homeless session... 2024-03-25T18:33:53.905 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.903+0000 7f70206e6640 10 client.4579.objecter successfully canceled tick 2024-03-25T18:33:53.905 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.903+0000 7f70206e6640 1 -- 172.21.15.3:0/1028316314 >> v1:172.21.15.3:6800/4195140966 conn(0x7f6ffc06e0b0 legacy=0x7f6ffc070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:53.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.903+0000 7f70206e6640 10 monclient: shutdown 2024-03-25T18:33:53.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.903+0000 7f70206e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:53.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.903+0000 7f70206e6640 1 -- 172.21.15.3:0/1028316314 >> v1:172.21.15.16:6789/0 conn(0x7f70181465b0 legacy=0x7f7018140f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:53.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.903+0000 7f70206e6640 1 -- 172.21.15.3:0/1028316314 shutdown_connections 2024-03-25T18:33:53.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.903+0000 7f70206e6640 1 -- 172.21.15.3:0/1028316314 >> 172.21.15.3:0/1028316314 conn(0x7f70180b5cf0 msgr2=0x7f70180b27f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:53.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.903+0000 7f70206e6640 1 -- 172.21.15.3:0/1028316314 shutdown_connections 2024-03-25T18:33:53.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.903+0000 7f70206e6640 1 -- 172.21.15.3:0/1028316314 wait complete. 2024-03-25T18:33:53.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:33:53.903+0000 7f70206e6640 1 librados: shutdown 2024-03-25T18:33:53.916 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:00.722 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:00.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.843+0000 7febe4f1c640 10 monclient: get_monmap_and_config 2024-03-25T18:34:00.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.843+0000 7febe4f1c640 10 monclient: build_initial_monmap 2024-03-25T18:34:00.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.843+0000 7febe4f1c640 10 monclient: monmap: 2024-03-25T18:34:00.845 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:34:00.845 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:00.845 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:34:00.844726+0000 2024-03-25T18:34:00.845 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:34:00.844726+0000 2024-03-25T18:34:00.845 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:00.845 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:00.845 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:34:00.845 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:34:00.845 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:34:00.845 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:00.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.844+0000 7febe4f1c640 1 Processor -- start 2024-03-25T18:34:00.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.844+0000 7febe4f1c640 1 -- start start 2024-03-25T18:34:00.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.844+0000 7febe4f1c640 10 monclient: init 2024-03-25T18:34:00.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.845+0000 7febe4f1c640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:00.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.845+0000 7febe4f1c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:34:00.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.845+0000 7febe4f1c640 10 monclient(hunting): picked mon.noname-b con 0x7febe00b4e70 addr v1:172.21.15.3:6790/0 2024-03-25T18:34:00.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.845+0000 7febe4f1c640 10 monclient(hunting): picked mon.noname-a con 0x7febe00b44b0 addr v1:172.21.15.3:6789/0 2024-03-25T18:34:00.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.845+0000 7febe4f1c640 10 monclient(hunting): picked mon.noname-c con 0x7febe00b1b00 addr v1:172.21.15.16:6789/0 2024-03-25T18:34:00.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.845+0000 7febe4f1c640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe00a96f0 con 0x7febe00b44b0 2024-03-25T18:34:00.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.845+0000 7febe4f1c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe00ab0e0 con 0x7febe00b1b00 2024-03-25T18:34:00.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.845+0000 7febe4f1c640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe0145f70 con 0x7febe00b4e70 2024-03-25T18:34:00.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.845+0000 7febe4f1c640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:00.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.845+0000 7febe4f1c640 10 monclient(hunting): authenticate will time out at 7895.105675s 2024-03-25T18:34:00.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.854+0000 7febde59c640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7febe00b4e70 0x7febe013f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:58932/0 (socket says 172.21.15.3:58932) 2024-03-25T18:34:00.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.854+0000 7febde59c640 1 -- 172.21.15.3:0/3875141532 learned_addr learned my addr 172.21.15.3:0/3875141532 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:34:00.863 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.862+0000 7febdd59a640 1 -- 172.21.15.3:0/3875141532 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2379937578 0 0) 0x7febe00a96f0 con 0x7febe00b44b0 2024-03-25T18:34:00.864 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.862+0000 7febdd59a640 10 monclient(hunting): _negotiate 2024-03-25T18:34:00.864 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.862+0000 7febdd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:00.864 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.862+0000 7febdd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:00.864 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.862+0000 7febdd59a640 10 monclient(hunting): my global_id is 4632 2024-03-25T18:34:00.864 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.862+0000 7febdd59a640 1 -- 172.21.15.3:0/3875141532 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febc8003520 con 0x7febe00b44b0 2024-03-25T18:34:00.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.875+0000 7febdd59a640 1 -- 172.21.15.3:0/3875141532 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4096516411 0 0) 0x7febe0145f70 con 0x7febe00b4e70 2024-03-25T18:34:00.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.875+0000 7febdd59a640 10 monclient(hunting): _negotiate 2024-03-25T18:34:00.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.875+0000 7febdd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:00.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.875+0000 7febdd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:00.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.875+0000 7febdd59a640 10 monclient(hunting): my global_id is 4586 2024-03-25T18:34:00.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.875+0000 7febdd59a640 1 -- 172.21.15.3:0/3875141532 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febe00a96f0 con 0x7febe00b4e70 2024-03-25T18:34:00.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.876+0000 7febdd59a640 1 -- 172.21.15.3:0/3875141532 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3455094834 0 0) 0x7febe00ab0e0 con 0x7febe00b1b00 2024-03-25T18:34:00.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.876+0000 7febdd59a640 10 monclient(hunting): _negotiate 2024-03-25T18:34:00.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.876+0000 7febdd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:00.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.876+0000 7febdd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:00.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.876+0000 7febdd59a640 10 monclient(hunting): my global_id is 4582 2024-03-25T18:34:00.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.876+0000 7febdd59a640 1 -- 172.21.15.3:0/3875141532 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febe0145f70 con 0x7febe00b1b00 2024-03-25T18:34:00.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.876+0000 7febdd59a640 1 -- 172.21.15.3:0/3875141532 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2352120644 0 0) 0x7febc8003520 con 0x7febe00b44b0 2024-03-25T18:34:00.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.876+0000 7febdd59a640 1 -- 172.21.15.3:0/3875141532 >> v1:172.21.15.3:6790/0 conn(0x7febe00b4e70 legacy=0x7febe013f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:00.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.876+0000 7febdd59a640 1 -- 172.21.15.3:0/3875141532 >> v1:172.21.15.16:6789/0 conn(0x7febe00b1b00 legacy=0x7febe014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:00.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.876+0000 7febdd59a640 10 monclient: _finish_hunting 0 2024-03-25T18:34:00.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.876+0000 7febdd59a640 1 monclient: found mon.noname-a 2024-03-25T18:34:00.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.876+0000 7febdd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:00.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.876+0000 7febdd59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:34:00.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.876+0000 7febdd59a640 1 -- 172.21.15.3:0/3875141532 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe00a9b80 con 0x7febe00b44b0 2024-03-25T18:34:00.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.876+0000 7febdd59a640 10 monclient: _finish_auth 0 2024-03-25T18:34:00.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.876+0000 7febdd59a640 10 monclient: _check_auth_tickets 2024-03-25T18:34:00.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.876+0000 7febdd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:00.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.876+0000 7febdd59a640 1 -- 172.21.15.3:0/3875141532 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7febd4001690 con 0x7febe00b44b0 2024-03-25T18:34:00.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.876+0000 7febe4f1c640 5 monclient: authenticate success, global_id 4632 2024-03-25T18:34:00.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.876+0000 7febe4f1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:00.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.876+0000 7febdd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:00.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.876+0000 7febdd59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:00.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.876+0000 7febdd59a640 10 monclient: dump: 2024-03-25T18:34:00.879 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:00.879 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:00.879 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:00.879 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:00.879 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:00.879 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:00.879 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:00.879 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:00.879 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:00.880 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:00.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.876+0000 7febe4f1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:00.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.883+0000 7febdd59a640 1 -- 172.21.15.3:0/3875141532 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febd40017f0 con 0x7febe00b44b0 2024-03-25T18:34:00.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.883+0000 7febdd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:00.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.883+0000 7febdd59a640 1 -- 172.21.15.3:0/3875141532 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7febd4001c00 con 0x7febe00b44b0 2024-03-25T18:34:00.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.883+0000 7febe4f1c640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:00.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.883+0000 7febe4f1c640 4 set_mon_vals no callback set 2024-03-25T18:34:00.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.883+0000 7febdd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:00.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.883+0000 7febdd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:00.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.883+0000 7febdd59a640 10 monclient: dump: 2024-03-25T18:34:00.885 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:00.885 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:00.885 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:00.885 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:00.885 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:00.885 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:00.885 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:00.886 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:00.886 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:00.886 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:00.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.884+0000 7febe4f1c640 10 monclient: shutdown 2024-03-25T18:34:00.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.884+0000 7febe4f1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:00.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.884+0000 7febe4f1c640 1 -- 172.21.15.3:0/3875141532 >> v1:172.21.15.3:6789/0 conn(0x7febe00b44b0 legacy=0x7febe0145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:00.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.884+0000 7febe4f1c640 1 -- 172.21.15.3:0/3875141532 shutdown_connections 2024-03-25T18:34:00.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.884+0000 7febe4f1c640 1 -- 172.21.15.3:0/3875141532 >> 172.21.15.3:0/3875141532 conn(0x7febe00617a0 msgr2=0x7febe0061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:00.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.884+0000 7febe4f1c640 1 -- 172.21.15.3:0/3875141532 shutdown_connections 2024-03-25T18:34:00.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.884+0000 7febe4f1c640 1 -- 172.21.15.3:0/3875141532 wait complete. 2024-03-25T18:34:00.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.885+0000 7febe4f1c640 10 monclient: build_initial_monmap 2024-03-25T18:34:00.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.885+0000 7febe4f1c640 10 monclient: monmap: 2024-03-25T18:34:00.889 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:34:00.889 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:00.889 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:34:00.889 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:34:00.889 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:00.889 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:34:00.889 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:34:00.889 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:34:00.889 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:34:00.890 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:00.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.885+0000 7febe4f1c640 1 librados: starting msgr at 2024-03-25T18:34:00.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.885+0000 7febe4f1c640 1 librados: starting objecter 2024-03-25T18:34:00.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.886+0000 7febe4f1c640 1 Processor -- start 2024-03-25T18:34:00.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.886+0000 7febe4f1c640 1 -- start start 2024-03-25T18:34:00.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.886+0000 7febe4f1c640 1 librados: setting wanted keys 2024-03-25T18:34:00.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.886+0000 7febe4f1c640 1 librados: calling monclient init 2024-03-25T18:34:00.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.886+0000 7febe4f1c640 10 monclient: init 2024-03-25T18:34:00.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.886+0000 7febe4f1c640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:00.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.886+0000 7febe4f1c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:34:00.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.886+0000 7febe4f1c640 10 monclient(hunting): picked mon.noname-b con 0x7febe00b1b00 addr v1:172.21.15.16:6789/0 2024-03-25T18:34:00.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.886+0000 7febe4f1c640 10 monclient(hunting): picked mon.noname-c con 0x7febe00b44b0 addr v1:172.21.15.3:6790/0 2024-03-25T18:34:00.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.887+0000 7febe4f1c640 10 monclient(hunting): picked mon.noname-a con 0x7febe00b4e70 addr v1:172.21.15.3:6789/0 2024-03-25T18:34:00.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.887+0000 7febe4f1c640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe0137f00 con 0x7febe00b4e70 2024-03-25T18:34:00.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.887+0000 7febe4f1c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe0136b00 con 0x7febe00b1b00 2024-03-25T18:34:00.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.887+0000 7febe4f1c640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe01d7b70 con 0x7febe00b44b0 2024-03-25T18:34:00.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.887+0000 7febe4f1c640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:00.891 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.887+0000 7febe4f1c640 10 monclient(hunting): authenticate will time out at 7895.147478s 2024-03-25T18:34:00.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.896+0000 7febded9d640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7febe00b4e70 0x7febe01d4c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:38872/0 (socket says 172.21.15.3:38872) 2024-03-25T18:34:00.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.896+0000 7febded9d640 1 -- 172.21.15.3:0/1130770657 learned_addr learned my addr 172.21.15.3:0/1130770657 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:34:00.898 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.897+0000 7febceffd640 10 client.?.objecter ms_handle_connect 0x7febe00b4e70 2024-03-25T18:34:00.898 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.897+0000 7febceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:00.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.905+0000 7febceffd640 1 -- 172.21.15.3:0/1130770657 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 277506932 0 0) 0x7febe0137f00 con 0x7febe00b4e70 2024-03-25T18:34:00.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.905+0000 7febceffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:00.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.905+0000 7febceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:00.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.905+0000 7febceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:00.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.905+0000 7febceffd640 10 monclient(hunting): my global_id is 4635 2024-03-25T18:34:00.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.905+0000 7febceffd640 1 -- 172.21.15.3:0/1130770657 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febc0004eb0 con 0x7febe00b4e70 2024-03-25T18:34:00.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.917+0000 7febceffd640 10 client.?.objecter ms_handle_connect 0x7febe00b44b0 2024-03-25T18:34:00.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.917+0000 7febceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:00.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.918+0000 7febceffd640 1 -- 172.21.15.3:0/1130770657 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 734494180 0 0) 0x7febc0004eb0 con 0x7febe00b4e70 2024-03-25T18:34:00.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.918+0000 7febceffd640 1 -- 172.21.15.3:0/1130770657 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febe0137f00 con 0x7febe00b4e70 2024-03-25T18:34:00.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.918+0000 7febceffd640 1 -- 172.21.15.3:0/1130770657 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7febd8001d10 con 0x7febe00b4e70 2024-03-25T18:34:00.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.918+0000 7febceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:00.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.918+0000 7febceffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:00.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.918+0000 7febceffd640 10 monclient(hunting): dump: 2024-03-25T18:34:00.920 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:00.920 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:00.920 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:00.920 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:00.920 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:00.920 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:00.920 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:00.920 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:00.920 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:00.920 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:00.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.918+0000 7febceffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:00.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.919+0000 7febceffd640 10 client.?.objecter ms_handle_connect 0x7febe00b1b00 2024-03-25T18:34:00.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.919+0000 7febceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:00.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.925+0000 7febceffd640 1 -- 172.21.15.3:0/1130770657 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 117364930 0 0) 0x7febe0137f00 con 0x7febe00b4e70 2024-03-25T18:34:00.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.925+0000 7febceffd640 1 -- 172.21.15.3:0/1130770657 >> v1:172.21.15.3:6790/0 conn(0x7febe00b44b0 legacy=0x7febe01d8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:00.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.925+0000 7febceffd640 1 -- 172.21.15.3:0/1130770657 >> v1:172.21.15.16:6789/0 conn(0x7febe00b1b00 legacy=0x7febe01363f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:00.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.925+0000 7febceffd640 10 monclient: _finish_hunting 0 2024-03-25T18:34:00.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.925+0000 7febceffd640 1 monclient: found mon.a 2024-03-25T18:34:00.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.925+0000 7febceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:00.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.925+0000 7febceffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:34:00.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.925+0000 7febceffd640 1 -- 172.21.15.3:0/1130770657 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe01d87a0 con 0x7febe00b4e70 2024-03-25T18:34:00.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.925+0000 7febceffd640 10 monclient: _finish_auth 0 2024-03-25T18:34:00.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.925+0000 7febceffd640 10 monclient: _check_auth_tickets 2024-03-25T18:34:00.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.925+0000 7febceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:00.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.925+0000 7febe4f1c640 5 monclient: authenticate success, global_id 4635 2024-03-25T18:34:00.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.925+0000 7febe4f1c640 10 monclient: _renew_subs 2024-03-25T18:34:00.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.925+0000 7febe4f1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:34:00.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.925+0000 7febe4f1c640 1 -- 172.21.15.3:0/1130770657 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe01d5360 con 0x7febe00b4e70 2024-03-25T18:34:00.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.925+0000 7febe4f1c640 10 client.4635.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:00.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.925+0000 7febe4f1c640 10 monclient: _renew_subs 2024-03-25T18:34:00.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.925+0000 7febe4f1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:34:00.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.925+0000 7febe4f1c640 1 -- 172.21.15.3:0/1130770657 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7febe01d4720 con 0x7febe00b4e70 2024-03-25T18:34:00.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.925+0000 7febe4f1c640 1 librados: init done 2024-03-25T18:34:00.929 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.927+0000 7febe4f1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:00.929 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.927+0000 7febe4f1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:00.929 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.927+0000 7febe4f1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:34:00.929 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:00.927+0000 7febe4f1c640 1 -- 172.21.15.3:0/1130770657 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe01d79d0 con 0x7febe00b4e70 2024-03-25T18:34:01.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.242+0000 7febceffd640 1 -- 172.21.15.3:0/1130770657 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febd8001ff0 con 0x7febe00b4e70 2024-03-25T18:34:01.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.242+0000 7febceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:01.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.242+0000 7febceffd640 1 -- 172.21.15.3:0/1130770657 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7febd8003c40 con 0x7febe00b4e70 2024-03-25T18:34:01.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.242+0000 7febceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:01.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.242+0000 7febceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:01.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.242+0000 7febceffd640 10 monclient: dump: 2024-03-25T18:34:01.244 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:01.244 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:01.244 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:01.244 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:01.244 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:01.245 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:01.245 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:01.245 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:01.245 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:01.245 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:01.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.242+0000 7febdcd99640 4 set_mon_vals no callback set 2024-03-25T18:34:01.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.243+0000 7febceffd640 1 -- 172.21.15.3:0/1130770657 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7febd8003e60 con 0x7febe00b4e70 2024-03-25T18:34:01.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.244+0000 7febde59c640 10 monclient: get_auth_request con 0x7febc006dc80 auth_method 0 2024-03-25T18:34:01.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.244+0000 7febceffd640 10 client.4635.objecter ms_handle_connect 0x7febc006dc80 2024-03-25T18:34:01.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.251+0000 7febceffd640 1 -- 172.21.15.3:0/1130770657 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7febd8054970 con 0x7febe00b4e70 2024-03-25T18:34:01.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.251+0000 7febceffd640 10 client.4635.objecter ms_dispatch 0x7febe0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:01.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.251+0000 7febceffd640 3 client.4635.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:01.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.251+0000 7febceffd640 3 client.4635.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:01.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.251+0000 7febceffd640 20 client.4635.objecter dump_active .. 0 homeless 2024-03-25T18:34:01.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.251+0000 7febceffd640 1 -- 172.21.15.3:0/1130770657 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7febd8050480 con 0x7febe00b4e70 2024-03-25T18:34:01.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.251+0000 7febceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:01.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.251+0000 7febceffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:01.509 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.508+0000 7febe4f1c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:01.509 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.508+0000 7febe4f1c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:01.509 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.508+0000 7febe4f1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:34:01.509 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.508+0000 7febe4f1c640 1 -- 172.21.15.3:0/1130770657 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febe01d7d00 con 0x7febe00b4e70 2024-03-25T18:34:01.530 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.529+0000 7febceffd640 1 -- 172.21.15.3:0/1130770657 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7febd80523b0 con 0x7febe00b4e70 2024-03-25T18:34:01.530 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.529+0000 7febceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:01.530 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.529+0000 7febceffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:01.531 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:34:01.531 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:01.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.531+0000 7febe4f1c640 10 librados: watch_flush enter 2024-03-25T18:34:01.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.531+0000 7febe4f1c640 10 librados: watch_flush exit 2024-03-25T18:34:01.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.531+0000 7febe4f1c640 20 client.4635.objecter shutdown clearing up homeless session... 2024-03-25T18:34:01.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.531+0000 7febe4f1c640 10 client.4635.objecter successfully canceled tick 2024-03-25T18:34:01.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.531+0000 7febe4f1c640 1 -- 172.21.15.3:0/1130770657 >> v1:172.21.15.3:6800/4195140966 conn(0x7febc006dc80 legacy=0x7febc0070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:01.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.531+0000 7febe4f1c640 10 monclient: shutdown 2024-03-25T18:34:01.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.531+0000 7febe4f1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:01.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.531+0000 7febe4f1c640 1 -- 172.21.15.3:0/1130770657 >> v1:172.21.15.3:6789/0 conn(0x7febe00b4e70 legacy=0x7febe01d4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:01.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.531+0000 7febe4f1c640 1 -- 172.21.15.3:0/1130770657 shutdown_connections 2024-03-25T18:34:01.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.531+0000 7febe4f1c640 1 -- 172.21.15.3:0/1130770657 >> 172.21.15.3:0/1130770657 conn(0x7febe00617a0 msgr2=0x7febe00b30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:01.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.531+0000 7febe4f1c640 1 -- 172.21.15.3:0/1130770657 shutdown_connections 2024-03-25T18:34:01.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.531+0000 7febe4f1c640 1 -- 172.21.15.3:0/1130770657 wait complete. 2024-03-25T18:34:01.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:01.531+0000 7febe4f1c640 1 librados: shutdown 2024-03-25T18:34:01.544 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:03.088 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:09:16.817 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:03.093 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:09:16.822 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:03.131 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:09:16.860 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:03.140 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:09:16.870 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:03.146 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:09:16.875 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:03.155 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:09:16.847 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:03.162 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:09:16.854 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:03.174 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:09:16.903 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:03.187 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:09:16.879 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:08.351 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:08.466 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.465+0000 7f1bd4ed3640 10 monclient: get_monmap_and_config 2024-03-25T18:34:08.466 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.465+0000 7f1bd4ed3640 10 monclient: build_initial_monmap 2024-03-25T18:34:08.467 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.465+0000 7f1bd4ed3640 10 monclient: monmap: 2024-03-25T18:34:08.467 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:34:08.467 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:08.467 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:34:08.466903+0000 2024-03-25T18:34:08.467 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:34:08.466903+0000 2024-03-25T18:34:08.467 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:08.467 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:08.467 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:34:08.467 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:34:08.467 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:34:08.467 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:08.469 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.466+0000 7f1bd4ed3640 1 Processor -- start 2024-03-25T18:34:08.469 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.466+0000 7f1bd4ed3640 1 -- start start 2024-03-25T18:34:08.469 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.466+0000 7f1bd4ed3640 10 monclient: init 2024-03-25T18:34:08.469 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.467+0000 7f1bd4ed3640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:08.469 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.467+0000 7f1bd4ed3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:34:08.469 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.467+0000 7f1bd4ed3640 10 monclient(hunting): picked mon.noname-b con 0x7f1bd00b4e70 addr v1:172.21.15.3:6790/0 2024-03-25T18:34:08.469 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.467+0000 7f1bd4ed3640 10 monclient(hunting): picked mon.noname-a con 0x7f1bd00b44b0 addr v1:172.21.15.3:6789/0 2024-03-25T18:34:08.469 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.467+0000 7f1bd4ed3640 10 monclient(hunting): picked mon.noname-c con 0x7f1bd00b1b00 addr v1:172.21.15.16:6789/0 2024-03-25T18:34:08.469 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.467+0000 7f1bd4ed3640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd00a96f0 con 0x7f1bd00b44b0 2024-03-25T18:34:08.469 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.467+0000 7f1bd4ed3640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd00ab0e0 con 0x7f1bd00b1b00 2024-03-25T18:34:08.469 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.467+0000 7f1bd4ed3640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd0145f70 con 0x7f1bd00b4e70 2024-03-25T18:34:08.470 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.467+0000 7f1bd4ed3640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:08.470 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.467+0000 7f1bd4ed3640 10 monclient(hunting): authenticate will time out at 7902.727864s 2024-03-25T18:34:08.472 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.471+0000 7f1bced9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1bd00b1b00 0x7f1bd014dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:45272/0 (socket says 172.21.15.3:45272) 2024-03-25T18:34:08.472 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.471+0000 7f1bced9d640 1 -- 172.21.15.3:0/2103994774 learned_addr learned my addr 172.21.15.3:0/2103994774 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:34:08.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.486+0000 7f1bcd59a640 1 -- 172.21.15.3:0/2103994774 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1720752799 0 0) 0x7f1bd0145f70 con 0x7f1bd00b4e70 2024-03-25T18:34:08.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.486+0000 7f1bcd59a640 10 monclient(hunting): _negotiate 2024-03-25T18:34:08.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.486+0000 7f1bcd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:08.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.486+0000 7f1bcd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:08.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.486+0000 7f1bcd59a640 10 monclient(hunting): my global_id is 4592 2024-03-25T18:34:08.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.486+0000 7f1bcd59a640 1 -- 172.21.15.3:0/2103994774 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bb8003520 con 0x7f1bd00b4e70 2024-03-25T18:34:08.491 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.490+0000 7f1bcd59a640 1 -- 172.21.15.3:0/2103994774 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2505793683 0 0) 0x7f1bd00ab0e0 con 0x7f1bd00b1b00 2024-03-25T18:34:08.491 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.490+0000 7f1bcd59a640 10 monclient(hunting): _negotiate 2024-03-25T18:34:08.491 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.490+0000 7f1bcd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:08.491 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.490+0000 7f1bcd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:08.491 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.490+0000 7f1bcd59a640 10 monclient(hunting): my global_id is 4588 2024-03-25T18:34:08.491 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.490+0000 7f1bcd59a640 1 -- 172.21.15.3:0/2103994774 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bd0145f70 con 0x7f1bd00b1b00 2024-03-25T18:34:08.505 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.504+0000 7f1bcd59a640 1 -- 172.21.15.3:0/2103994774 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1266997165 0 0) 0x7f1bd0145f70 con 0x7f1bd00b1b00 2024-03-25T18:34:08.505 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.504+0000 7f1bcd59a640 1 -- 172.21.15.3:0/2103994774 >> v1:172.21.15.3:6790/0 conn(0x7f1bd00b4e70 legacy=0x7f1bd013f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:08.505 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.504+0000 7f1bcd59a640 1 -- 172.21.15.3:0/2103994774 >> v1:172.21.15.3:6789/0 conn(0x7f1bd00b44b0 legacy=0x7f1bd0145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:08.505 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.504+0000 7f1bcd59a640 10 monclient: _finish_hunting 0 2024-03-25T18:34:08.505 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.504+0000 7f1bcd59a640 1 monclient: found mon.noname-c 2024-03-25T18:34:08.505 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.504+0000 7f1bcd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:08.505 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.504+0000 7f1bcd59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:34:08.505 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.504+0000 7f1bcd59a640 1 -- 172.21.15.3:0/2103994774 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd00a9b80 con 0x7f1bd00b1b00 2024-03-25T18:34:08.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.504+0000 7f1bcd59a640 10 monclient: _finish_auth 0 2024-03-25T18:34:08.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.504+0000 7f1bcd59a640 10 monclient: _check_auth_tickets 2024-03-25T18:34:08.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.504+0000 7f1bcd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:08.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.504+0000 7f1bcd59a640 1 -- 172.21.15.3:0/2103994774 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f1bc80019f0 con 0x7f1bd00b1b00 2024-03-25T18:34:08.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.504+0000 7f1bcd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:08.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.504+0000 7f1bcd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:08.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.504+0000 7f1bcd59a640 10 monclient: dump: 2024-03-25T18:34:08.506 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:08.506 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:08.506 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:08.506 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:08.506 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:08.506 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:08.506 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:08.506 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:08.507 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:08.507 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:08.507 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.504+0000 7f1bd4ed3640 5 monclient: authenticate success, global_id 4588 2024-03-25T18:34:08.507 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.504+0000 7f1bd4ed3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:08.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.536+0000 7f1bcd59a640 1 -- 172.21.15.3:0/2103994774 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bc8001b50 con 0x7f1bd00b1b00 2024-03-25T18:34:08.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.536+0000 7f1bcd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:08.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.536+0000 7f1bcd59a640 1 -- 172.21.15.3:0/2103994774 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f1bc8001f20 con 0x7f1bd00b1b00 2024-03-25T18:34:08.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.536+0000 7f1bd4ed3640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:08.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.536+0000 7f1bd4ed3640 4 set_mon_vals no callback set 2024-03-25T18:34:08.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.536+0000 7f1bcd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:08.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.536+0000 7f1bcd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:08.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.536+0000 7f1bcd59a640 10 monclient: dump: 2024-03-25T18:34:08.537 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:08.537 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:08.538 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:08.538 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:08.538 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:08.538 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:08.538 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:08.538 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:08.538 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:08.538 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:08.540 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.537+0000 7f1bd4ed3640 10 monclient: shutdown 2024-03-25T18:34:08.540 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.537+0000 7f1bd4ed3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:08.540 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.537+0000 7f1bd4ed3640 1 -- 172.21.15.3:0/2103994774 >> v1:172.21.15.16:6789/0 conn(0x7f1bd00b1b00 legacy=0x7f1bd014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:08.540 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.537+0000 7f1bd4ed3640 1 -- 172.21.15.3:0/2103994774 shutdown_connections 2024-03-25T18:34:08.540 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.537+0000 7f1bd4ed3640 1 -- 172.21.15.3:0/2103994774 >> 172.21.15.3:0/2103994774 conn(0x7f1bd00617a0 msgr2=0x7f1bd0061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:08.540 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.537+0000 7f1bd4ed3640 1 -- 172.21.15.3:0/2103994774 shutdown_connections 2024-03-25T18:34:08.540 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.537+0000 7f1bd4ed3640 1 -- 172.21.15.3:0/2103994774 wait complete. 2024-03-25T18:34:08.540 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.538+0000 7f1bd4ed3640 10 monclient: build_initial_monmap 2024-03-25T18:34:08.540 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.538+0000 7f1bd4ed3640 10 monclient: monmap: 2024-03-25T18:34:08.540 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:34:08.541 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:08.541 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:34:08.541 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:34:08.541 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:08.541 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:34:08.541 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:34:08.541 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:34:08.541 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:34:08.541 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:08.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.538+0000 7f1bd4ed3640 1 librados: starting msgr at 2024-03-25T18:34:08.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.538+0000 7f1bd4ed3640 1 librados: starting objecter 2024-03-25T18:34:08.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.539+0000 7f1bd4ed3640 1 Processor -- start 2024-03-25T18:34:08.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.539+0000 7f1bd4ed3640 1 -- start start 2024-03-25T18:34:08.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.539+0000 7f1bd4ed3640 1 librados: setting wanted keys 2024-03-25T18:34:08.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.539+0000 7f1bd4ed3640 1 librados: calling monclient init 2024-03-25T18:34:08.542 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.539+0000 7f1bd4ed3640 10 monclient: init 2024-03-25T18:34:08.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.539+0000 7f1bd4ed3640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:08.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.539+0000 7f1bd4ed3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:34:08.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.539+0000 7f1bd4ed3640 10 monclient(hunting): picked mon.noname-c con 0x7f1bd00b1b00 addr v1:172.21.15.3:6790/0 2024-03-25T18:34:08.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.539+0000 7f1bd4ed3640 10 monclient(hunting): picked mon.noname-b con 0x7f1bd00b44b0 addr v1:172.21.15.16:6789/0 2024-03-25T18:34:08.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.539+0000 7f1bd4ed3640 10 monclient(hunting): picked mon.noname-a con 0x7f1bd00b4e70 addr v1:172.21.15.3:6789/0 2024-03-25T18:34:08.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.539+0000 7f1bd4ed3640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd0137fa0 con 0x7f1bd00b4e70 2024-03-25T18:34:08.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.540+0000 7f1bd4ed3640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd01d7900 con 0x7f1bd00b44b0 2024-03-25T18:34:08.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.540+0000 7f1bd4ed3640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd0136df0 con 0x7f1bd00b1b00 2024-03-25T18:34:08.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.540+0000 7f1bd4ed3640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:08.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.540+0000 7f1bd4ed3640 10 monclient(hunting): authenticate will time out at 7902.800366s 2024-03-25T18:34:08.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.553+0000 7f1bcdd9b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1bd00b44b0 0x7f1bd01d7e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:59024/0 (socket says 172.21.15.3:59024) 2024-03-25T18:34:08.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.553+0000 7f1bcdd9b640 1 -- 172.21.15.3:0/3851034277 learned_addr learned my addr 172.21.15.3:0/3851034277 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:34:08.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.555+0000 7f1bbeffd640 10 client.?.objecter ms_handle_connect 0x7f1bd00b44b0 2024-03-25T18:34:08.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.555+0000 7f1bbeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:08.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.558+0000 7f1bbeffd640 10 client.?.objecter ms_handle_connect 0x7f1bd00b4e70 2024-03-25T18:34:08.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.558+0000 7f1bbeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:08.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.559+0000 7f1bbeffd640 10 client.?.objecter ms_handle_connect 0x7f1bd00b1b00 2024-03-25T18:34:08.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.559+0000 7f1bbeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:08.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.564+0000 7f1bbeffd640 1 -- 172.21.15.3:0/3851034277 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1860330603 0 0) 0x7f1bd01d7900 con 0x7f1bd00b44b0 2024-03-25T18:34:08.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.564+0000 7f1bbeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:08.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.564+0000 7f1bbeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:08.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.564+0000 7f1bbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:08.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.564+0000 7f1bbeffd640 10 monclient(hunting): my global_id is 4591 2024-03-25T18:34:08.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.564+0000 7f1bbeffd640 1 -- 172.21.15.3:0/3851034277 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bb0004eb0 con 0x7f1bd00b44b0 2024-03-25T18:34:08.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.576+0000 7f1bbeffd640 1 -- 172.21.15.3:0/3851034277 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2731963377 0 0) 0x7f1bb0004eb0 con 0x7f1bd00b44b0 2024-03-25T18:34:08.578 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.576+0000 7f1bbeffd640 1 -- 172.21.15.3:0/3851034277 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1bd01d7900 con 0x7f1bd00b44b0 2024-03-25T18:34:08.578 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.576+0000 7f1bbeffd640 1 -- 172.21.15.3:0/3851034277 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f1bc4001970 con 0x7f1bd00b44b0 2024-03-25T18:34:08.578 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.576+0000 7f1bbeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:08.578 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.576+0000 7f1bbeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:08.578 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.576+0000 7f1bbeffd640 10 monclient(hunting): dump: 2024-03-25T18:34:08.578 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:08.578 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:08.578 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:08.578 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:08.578 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:08.578 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:08.578 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:08.578 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:08.579 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:08.579 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:08.579 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.576+0000 7f1bbeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:08.583 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.582+0000 7f1bbeffd640 1 -- 172.21.15.3:0/3851034277 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 73256179 0 0) 0x7f1bd0136df0 con 0x7f1bd00b1b00 2024-03-25T18:34:08.583 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.582+0000 7f1bbeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:08.583 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.582+0000 7f1bbeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:08.583 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.582+0000 7f1bbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:08.583 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.582+0000 7f1bbeffd640 10 monclient(hunting): my global_id is 4595 2024-03-25T18:34:08.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.582+0000 7f1bbeffd640 1 -- 172.21.15.3:0/3851034277 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bb0004eb0 con 0x7f1bd00b1b00 2024-03-25T18:34:08.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.591+0000 7f1bbeffd640 1 -- 172.21.15.3:0/3851034277 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 273613314 0 0) 0x7f1bd0137fa0 con 0x7f1bd00b4e70 2024-03-25T18:34:08.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.591+0000 7f1bbeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:08.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.591+0000 7f1bbeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:08.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.591+0000 7f1bbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:08.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.591+0000 7f1bbeffd640 10 monclient(hunting): my global_id is 4641 2024-03-25T18:34:08.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.591+0000 7f1bbeffd640 1 -- 172.21.15.3:0/3851034277 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bd0136df0 con 0x7f1bd00b4e70 2024-03-25T18:34:08.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.597+0000 7f1bbeffd640 1 -- 172.21.15.3:0/3851034277 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1863276729 0 0) 0x7f1bd01d7900 con 0x7f1bd00b44b0 2024-03-25T18:34:08.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.597+0000 7f1bbeffd640 1 -- 172.21.15.3:0/3851034277 >> v1:172.21.15.3:6790/0 conn(0x7f1bd00b1b00 legacy=0x7f1bd01366e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:08.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.597+0000 7f1bbeffd640 1 -- 172.21.15.3:0/3851034277 >> v1:172.21.15.3:6789/0 conn(0x7f1bd00b4e70 legacy=0x7f1bd01e22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:08.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.597+0000 7f1bbeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:34:08.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.597+0000 7f1bbeffd640 1 monclient: found mon.b 2024-03-25T18:34:08.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.597+0000 7f1bbeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:08.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.597+0000 7f1bbeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:34:08.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.597+0000 7f1bbeffd640 1 -- 172.21.15.3:0/3851034277 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd0136f80 con 0x7f1bd00b44b0 2024-03-25T18:34:08.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.597+0000 7f1bbeffd640 10 monclient: _finish_auth 0 2024-03-25T18:34:08.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.597+0000 7f1bbeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:34:08.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.597+0000 7f1bbeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:08.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.597+0000 7f1bd4ed3640 5 monclient: authenticate success, global_id 4591 2024-03-25T18:34:08.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.597+0000 7f1bd4ed3640 10 monclient: _renew_subs 2024-03-25T18:34:08.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.597+0000 7f1bd4ed3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:34:08.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.597+0000 7f1bd4ed3640 1 -- 172.21.15.3:0/3851034277 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bd0141750 con 0x7f1bd00b44b0 2024-03-25T18:34:08.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.597+0000 7f1bd4ed3640 10 client.4591.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:08.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.597+0000 7f1bd4ed3640 10 monclient: _renew_subs 2024-03-25T18:34:08.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.597+0000 7f1bd4ed3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:34:08.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.597+0000 7f1bd4ed3640 1 -- 172.21.15.3:0/3851034277 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bd01d8710 con 0x7f1bd00b44b0 2024-03-25T18:34:08.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.597+0000 7f1bd4ed3640 1 librados: init done 2024-03-25T18:34:08.601 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.598+0000 7f1bd4ed3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:08.601 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.598+0000 7f1bd4ed3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:08.601 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.598+0000 7f1bd4ed3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:34:08.601 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.598+0000 7f1bd4ed3640 1 -- 172.21.15.3:0/3851034277 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bd01d7a90 con 0x7f1bd00b44b0 2024-03-25T18:34:08.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.887+0000 7f1bbeffd640 1 -- 172.21.15.3:0/3851034277 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bc4003610 con 0x7f1bd00b44b0 2024-03-25T18:34:08.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.887+0000 7f1bbeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:08.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.887+0000 7f1bbeffd640 1 -- 172.21.15.3:0/3851034277 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f1bc4003960 con 0x7f1bd00b44b0 2024-03-25T18:34:08.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.887+0000 7f1bbeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:08.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.887+0000 7f1bbeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:08.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.887+0000 7f1bbeffd640 10 monclient: dump: 2024-03-25T18:34:08.889 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:08.889 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:08.889 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:08.889 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:08.889 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:08.889 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:08.889 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:08.889 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:08.889 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:08.890 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:08.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.887+0000 7f1bccd99640 4 set_mon_vals no callback set 2024-03-25T18:34:08.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.888+0000 7f1bbeffd640 1 -- 172.21.15.3:0/3851034277 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f1bc4003b80 con 0x7f1bd00b44b0 2024-03-25T18:34:08.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.888+0000 7f1bce59c640 10 monclient: get_auth_request con 0x7f1bb006de60 auth_method 0 2024-03-25T18:34:08.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.889+0000 7f1bbeffd640 10 client.4591.objecter ms_handle_connect 0x7f1bb006de60 2024-03-25T18:34:08.896 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.895+0000 7f1bbeffd640 1 -- 172.21.15.3:0/3851034277 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f1bc4054790 con 0x7f1bd00b44b0 2024-03-25T18:34:08.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.895+0000 7f1bbeffd640 10 client.4591.objecter ms_dispatch 0x7f1bd0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:08.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.895+0000 7f1bbeffd640 3 client.4591.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:08.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.895+0000 7f1bbeffd640 3 client.4591.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:08.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.895+0000 7f1bbeffd640 20 client.4591.objecter dump_active .. 0 homeless 2024-03-25T18:34:08.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.896+0000 7f1bbeffd640 1 -- 172.21.15.3:0/3851034277 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1bc40502a0 con 0x7f1bd00b44b0 2024-03-25T18:34:08.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.896+0000 7f1bbeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:08.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:08.896+0000 7f1bbeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:09.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:09.154+0000 7f1bd4ed3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:09.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:09.154+0000 7f1bd4ed3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:09.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:09.154+0000 7f1bd4ed3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:34:09.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:09.154+0000 7f1bd4ed3640 1 -- 172.21.15.3:0/3851034277 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1bd004f730 con 0x7f1bd00b44b0 2024-03-25T18:34:09.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:09.175+0000 7f1bbeffd640 1 -- 172.21.15.3:0/3851034277 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f1bc40521d0 con 0x7f1bd00b44b0 2024-03-25T18:34:09.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:09.175+0000 7f1bbeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:09.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:09.175+0000 7f1bbeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:09.177 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:34:09.177 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:09.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:09.178+0000 7f1bd4ed3640 10 librados: watch_flush enter 2024-03-25T18:34:09.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:09.178+0000 7f1bd4ed3640 10 librados: watch_flush exit 2024-03-25T18:34:09.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:09.178+0000 7f1bd4ed3640 20 client.4591.objecter shutdown clearing up homeless session... 2024-03-25T18:34:09.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:09.178+0000 7f1bd4ed3640 10 client.4591.objecter successfully canceled tick 2024-03-25T18:34:09.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:09.178+0000 7f1bd4ed3640 1 -- 172.21.15.3:0/3851034277 >> v1:172.21.15.3:6800/4195140966 conn(0x7f1bb006de60 legacy=0x7f1bb0070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:09.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:09.178+0000 7f1bd4ed3640 10 monclient: shutdown 2024-03-25T18:34:09.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:09.178+0000 7f1bd4ed3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:09.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:09.178+0000 7f1bd4ed3640 1 -- 172.21.15.3:0/3851034277 >> v1:172.21.15.16:6789/0 conn(0x7f1bd00b44b0 legacy=0x7f1bd01d7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:09.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:09.178+0000 7f1bd4ed3640 1 -- 172.21.15.3:0/3851034277 shutdown_connections 2024-03-25T18:34:09.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:09.178+0000 7f1bd4ed3640 1 -- 172.21.15.3:0/3851034277 >> 172.21.15.3:0/3851034277 conn(0x7f1bd00617a0 msgr2=0x7f1bd0144a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:09.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:09.178+0000 7f1bd4ed3640 1 -- 172.21.15.3:0/3851034277 shutdown_connections 2024-03-25T18:34:09.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:09.178+0000 7f1bd4ed3640 1 -- 172.21.15.3:0/3851034277 wait complete. 2024-03-25T18:34:09.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:09.178+0000 7f1bd4ed3640 1 librados: shutdown 2024-03-25T18:34:09.190 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:15.996 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:16.093 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.093+0000 7f6742fd4640 10 monclient: get_monmap_and_config 2024-03-25T18:34:16.093 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.093+0000 7f6742fd4640 10 monclient: build_initial_monmap 2024-03-25T18:34:16.093 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.093+0000 7f6742fd4640 10 monclient: monmap: 2024-03-25T18:34:16.093 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:34:16.093 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:16.093 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:34:16.093288+0000 2024-03-25T18:34:16.093 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:34:16.093288+0000 2024-03-25T18:34:16.093 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:16.093 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:16.093 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:34:16.094 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:34:16.094 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:34:16.094 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:16.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.093+0000 7f6742fd4640 1 Processor -- start 2024-03-25T18:34:16.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.093+0000 7f6742fd4640 1 -- start start 2024-03-25T18:34:16.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.093+0000 7f6742fd4640 10 monclient: init 2024-03-25T18:34:16.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.094+0000 7f6742fd4640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:16.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.094+0000 7f6742fd4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:34:16.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.094+0000 7f6742fd4640 10 monclient(hunting): picked mon.noname-c con 0x7f673c0b4e70 addr v1:172.21.15.16:6789/0 2024-03-25T18:34:16.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.094+0000 7f6742fd4640 10 monclient(hunting): picked mon.noname-b con 0x7f673c0b44b0 addr v1:172.21.15.3:6790/0 2024-03-25T18:34:16.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.094+0000 7f6742fd4640 10 monclient(hunting): picked mon.noname-a con 0x7f673c0b1b00 addr v1:172.21.15.3:6789/0 2024-03-25T18:34:16.096 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.094+0000 7f6742fd4640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f673c0a96f0 con 0x7f673c0b1b00 2024-03-25T18:34:16.096 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.094+0000 7f6742fd4640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f673c0ab0e0 con 0x7f673c0b4e70 2024-03-25T18:34:16.096 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.094+0000 7f6742fd4640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f673c145f70 con 0x7f673c0b44b0 2024-03-25T18:34:16.096 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.094+0000 7f6742fd4640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:16.096 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.094+0000 7f6742fd4640 10 monclient(hunting): authenticate will time out at 7910.353962s 2024-03-25T18:34:16.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.106+0000 7f6741571640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f673c0b1b00 0x7f673c14dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:36330/0 (socket says 172.21.15.3:36330) 2024-03-25T18:34:16.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.106+0000 7f6741571640 1 -- 172.21.15.3:0/3448623006 learned_addr learned my addr 172.21.15.3:0/3448623006 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:34:16.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.124+0000 7f673b7fe640 1 -- 172.21.15.3:0/3448623006 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2827086327 0 0) 0x7f673c0a96f0 con 0x7f673c0b1b00 2024-03-25T18:34:16.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.124+0000 7f673b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:16.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.124+0000 7f673b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:16.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.124+0000 7f673b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:16.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.124+0000 7f673b7fe640 10 monclient(hunting): my global_id is 4644 2024-03-25T18:34:16.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.124+0000 7f673b7fe640 1 -- 172.21.15.3:0/3448623006 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f672c003520 con 0x7f673c0b1b00 2024-03-25T18:34:16.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.152+0000 7f673b7fe640 1 -- 172.21.15.3:0/3448623006 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2934170087 0 0) 0x7f672c003520 con 0x7f673c0b1b00 2024-03-25T18:34:16.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.152+0000 7f673b7fe640 1 -- 172.21.15.3:0/3448623006 >> v1:172.21.15.3:6790/0 conn(0x7f673c0b44b0 legacy=0x7f673c145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:34:16.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.152+0000 7f673b7fe640 1 -- 172.21.15.3:0/3448623006 >> v1:172.21.15.16:6789/0 conn(0x7f673c0b4e70 legacy=0x7f673c13f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:34:16.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.152+0000 7f673b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:34:16.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.152+0000 7f673b7fe640 1 monclient: found mon.noname-a 2024-03-25T18:34:16.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.152+0000 7f673b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:16.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.152+0000 7f673b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:34:16.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.152+0000 7f673b7fe640 1 -- 172.21.15.3:0/3448623006 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f673c0a9b80 con 0x7f673c0b1b00 2024-03-25T18:34:16.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.152+0000 7f673b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:34:16.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.152+0000 7f673b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:34:16.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.152+0000 7f673b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:16.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.152+0000 7f673b7fe640 1 -- 172.21.15.3:0/3448623006 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f6734001a10 con 0x7f673c0b1b00 2024-03-25T18:34:16.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.152+0000 7f6742fd4640 5 monclient: authenticate success, global_id 4644 2024-03-25T18:34:16.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.152+0000 7f6742fd4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:16.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.152+0000 7f673b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:16.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.152+0000 7f673b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:16.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.152+0000 7f673b7fe640 10 monclient: dump: 2024-03-25T18:34:16.154 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:16.154 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:16.154 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:16.154 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:16.154 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:16.154 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:16.154 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:16.154 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:16.154 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:16.154 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:16.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.152+0000 7f6742fd4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:16.176 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.176+0000 7f673b7fe640 1 -- 172.21.15.3:0/3448623006 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6734001be0 con 0x7f673c0b1b00 2024-03-25T18:34:16.176 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.176+0000 7f673b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:16.176 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.176+0000 7f673b7fe640 1 -- 172.21.15.3:0/3448623006 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f6734002030 con 0x7f673c0b1b00 2024-03-25T18:34:16.176 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.176+0000 7f673b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:16.176 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.176+0000 7f673b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:16.176 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.176+0000 7f673b7fe640 10 monclient: dump: 2024-03-25T18:34:16.176 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:16.176 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:16.176 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:16.176 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:16.177 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:16.177 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:16.177 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:16.177 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:16.177 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:16.177 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:16.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.176+0000 7f6742fd4640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:16.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.176+0000 7f6742fd4640 4 set_mon_vals no callback set 2024-03-25T18:34:16.179 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.176+0000 7f6742fd4640 10 monclient: shutdown 2024-03-25T18:34:16.179 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.176+0000 7f6742fd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:16.179 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.176+0000 7f6742fd4640 1 -- 172.21.15.3:0/3448623006 >> v1:172.21.15.3:6789/0 conn(0x7f673c0b1b00 legacy=0x7f673c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:16.179 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.176+0000 7f6742fd4640 1 -- 172.21.15.3:0/3448623006 shutdown_connections 2024-03-25T18:34:16.179 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.176+0000 7f6742fd4640 1 -- 172.21.15.3:0/3448623006 >> 172.21.15.3:0/3448623006 conn(0x7f673c0617a0 msgr2=0x7f673c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:16.179 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.177+0000 7f6742fd4640 1 -- 172.21.15.3:0/3448623006 shutdown_connections 2024-03-25T18:34:16.179 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.177+0000 7f6742fd4640 1 -- 172.21.15.3:0/3448623006 wait complete. 2024-03-25T18:34:16.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.177+0000 7f6742fd4640 10 monclient: build_initial_monmap 2024-03-25T18:34:16.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.177+0000 7f6742fd4640 10 monclient: monmap: 2024-03-25T18:34:16.180 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:34:16.180 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:16.180 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:34:16.180 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:34:16.180 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:16.180 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:34:16.180 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:34:16.180 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:34:16.180 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:34:16.180 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:16.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.178+0000 7f6742fd4640 1 librados: starting msgr at 2024-03-25T18:34:16.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.178+0000 7f6742fd4640 1 librados: starting objecter 2024-03-25T18:34:16.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.178+0000 7f6742fd4640 1 Processor -- start 2024-03-25T18:34:16.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.178+0000 7f6742fd4640 1 -- start start 2024-03-25T18:34:16.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.178+0000 7f6742fd4640 1 librados: setting wanted keys 2024-03-25T18:34:16.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.178+0000 7f6742fd4640 1 librados: calling monclient init 2024-03-25T18:34:16.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.178+0000 7f6742fd4640 10 monclient: init 2024-03-25T18:34:16.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.179+0000 7f6742fd4640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:16.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.179+0000 7f6742fd4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:34:16.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.179+0000 7f6742fd4640 10 monclient(hunting): picked mon.noname-b con 0x7f673c0b1b00 addr v1:172.21.15.16:6789/0 2024-03-25T18:34:16.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.179+0000 7f6742fd4640 10 monclient(hunting): picked mon.noname-a con 0x7f673c0b44b0 addr v1:172.21.15.3:6789/0 2024-03-25T18:34:16.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.179+0000 7f6742fd4640 10 monclient(hunting): picked mon.noname-c con 0x7f673c0b4e70 addr v1:172.21.15.3:6790/0 2024-03-25T18:34:16.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.179+0000 7f6742fd4640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f673c142610 con 0x7f673c0b44b0 2024-03-25T18:34:16.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.179+0000 7f6742fd4640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f673c1430f0 con 0x7f673c0b1b00 2024-03-25T18:34:16.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.179+0000 7f6742fd4640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f673c1433d0 con 0x7f673c0b4e70 2024-03-25T18:34:16.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.179+0000 7f6742fd4640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:16.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.179+0000 7f6742fd4640 10 monclient(hunting): authenticate will time out at 7910.438725s 2024-03-25T18:34:16.182 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.180+0000 7f6740d70640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f673c0b1b00 0x7f673c133e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:35336/0 (socket says 172.21.15.3:35336) 2024-03-25T18:34:16.182 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.180+0000 7f6740d70640 1 -- 172.21.15.3:0/2702382432 learned_addr learned my addr 172.21.15.3:0/2702382432 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:34:16.182 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.181+0000 7f67397fa640 10 client.?.objecter ms_handle_connect 0x7f673c0b1b00 2024-03-25T18:34:16.182 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.181+0000 7f67397fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:16.182 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.181+0000 7f67397fa640 10 client.?.objecter ms_handle_connect 0x7f673c0b4e70 2024-03-25T18:34:16.182 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.181+0000 7f67397fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:16.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.242+0000 7f67397fa640 1 -- 172.21.15.3:0/2702382432 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1204076346 0 0) 0x7f673c1430f0 con 0x7f673c0b1b00 2024-03-25T18:34:16.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.242+0000 7f67397fa640 10 monclient(hunting): _negotiate 2024-03-25T18:34:16.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.242+0000 7f67397fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:16.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.242+0000 7f67397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:16.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.242+0000 7f67397fa640 10 monclient(hunting): my global_id is 4594 2024-03-25T18:34:16.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.242+0000 7f67397fa640 1 -- 172.21.15.3:0/2702382432 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6720004eb0 con 0x7f673c0b1b00 2024-03-25T18:34:16.261 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.261+0000 7f67397fa640 10 client.?.objecter ms_handle_connect 0x7f673c0b44b0 2024-03-25T18:34:16.261 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.261+0000 7f67397fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:16.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.277+0000 7f67397fa640 1 -- 172.21.15.3:0/2702382432 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1423572614 0 0) 0x7f6720004eb0 con 0x7f673c0b1b00 2024-03-25T18:34:16.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.277+0000 7f67397fa640 1 -- 172.21.15.3:0/2702382432 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f673c1430f0 con 0x7f673c0b1b00 2024-03-25T18:34:16.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.277+0000 7f67397fa640 1 -- 172.21.15.3:0/2702382432 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f67300017d0 con 0x7f673c0b1b00 2024-03-25T18:34:16.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.277+0000 7f67397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:16.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.277+0000 7f67397fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:16.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.277+0000 7f67397fa640 10 monclient(hunting): dump: 2024-03-25T18:34:16.278 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:16.278 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:16.278 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:16.278 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:16.278 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:16.278 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:16.279 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:16.279 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:16.279 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:16.279 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:16.279 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.277+0000 7f67397fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:16.279 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.278+0000 7f67397fa640 1 -- 172.21.15.3:0/2702382432 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2535944105 0 0) 0x7f673c142610 con 0x7f673c0b44b0 2024-03-25T18:34:16.279 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.278+0000 7f67397fa640 10 monclient(hunting): _negotiate 2024-03-25T18:34:16.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.278+0000 7f67397fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:16.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.278+0000 7f67397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:16.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.278+0000 7f67397fa640 10 monclient(hunting): my global_id is 4647 2024-03-25T18:34:16.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.278+0000 7f67397fa640 1 -- 172.21.15.3:0/2702382432 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6720004eb0 con 0x7f673c0b44b0 2024-03-25T18:34:16.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.286+0000 7f67397fa640 1 -- 172.21.15.3:0/2702382432 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2558313163 0 0) 0x7f673c1430f0 con 0x7f673c0b1b00 2024-03-25T18:34:16.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.286+0000 7f67397fa640 1 -- 172.21.15.3:0/2702382432 >> v1:172.21.15.3:6790/0 conn(0x7f673c0b4e70 legacy=0x7f673c1e2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:16.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.286+0000 7f67397fa640 1 -- 172.21.15.3:0/2702382432 >> v1:172.21.15.3:6789/0 conn(0x7f673c0b44b0 legacy=0x7f673c1dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:16.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.286+0000 7f67397fa640 10 monclient: _finish_hunting 0 2024-03-25T18:34:16.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.286+0000 7f67397fa640 1 monclient: found mon.b 2024-03-25T18:34:16.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.286+0000 7f67397fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:16.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.286+0000 7f67397fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:34:16.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.286+0000 7f67397fa640 1 -- 172.21.15.3:0/2702382432 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f673c1420f0 con 0x7f673c0b1b00 2024-03-25T18:34:16.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.286+0000 7f67397fa640 10 monclient: _finish_auth 0 2024-03-25T18:34:16.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.286+0000 7f67397fa640 10 monclient: _check_auth_tickets 2024-03-25T18:34:16.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.286+0000 7f67397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:16.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.286+0000 7f6742fd4640 5 monclient: authenticate success, global_id 4594 2024-03-25T18:34:16.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.286+0000 7f6742fd4640 10 monclient: _renew_subs 2024-03-25T18:34:16.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.286+0000 7f6742fd4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:34:16.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.286+0000 7f6742fd4640 1 -- 172.21.15.3:0/2702382432 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f673c1412c0 con 0x7f673c0b1b00 2024-03-25T18:34:16.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.286+0000 7f6742fd4640 10 client.4594.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:16.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.286+0000 7f6742fd4640 10 monclient: _renew_subs 2024-03-25T18:34:16.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.286+0000 7f6742fd4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:34:16.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.286+0000 7f6742fd4640 1 -- 172.21.15.3:0/2702382432 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f673c134590 con 0x7f673c0b1b00 2024-03-25T18:34:16.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.286+0000 7f6742fd4640 1 librados: init done 2024-03-25T18:34:16.288 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.287+0000 7f6742fd4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:16.288 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.287+0000 7f6742fd4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:16.288 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.287+0000 7f6742fd4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:34:16.288 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.287+0000 7f6742fd4640 1 -- 172.21.15.3:0/2702382432 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f673c04f730 con 0x7f673c0b1b00 2024-03-25T18:34:16.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.562+0000 7f67397fa640 1 -- 172.21.15.3:0/2702382432 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6730001d20 con 0x7f673c0b1b00 2024-03-25T18:34:16.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.563+0000 7f67397fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:16.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.563+0000 7f67397fa640 1 -- 172.21.15.3:0/2702382432 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f6730004a00 con 0x7f673c0b1b00 2024-03-25T18:34:16.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.563+0000 7f67397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:16.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.563+0000 7f67397fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:16.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.563+0000 7f67397fa640 10 monclient: dump: 2024-03-25T18:34:16.563 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:16.563 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:16.563 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:16.563 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:16.563 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:16.564 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:16.564 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:16.564 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:16.564 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:16.564 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:16.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.563+0000 7f673affd640 4 set_mon_vals no callback set 2024-03-25T18:34:16.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.563+0000 7f67397fa640 1 -- 172.21.15.3:0/2702382432 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f673001a9d0 con 0x7f673c0b1b00 2024-03-25T18:34:16.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.564+0000 7f673bfff640 10 monclient: get_auth_request con 0x7f672006dd00 auth_method 0 2024-03-25T18:34:16.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.564+0000 7f67397fa640 10 client.4594.objecter ms_handle_connect 0x7f672006dd00 2024-03-25T18:34:16.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.571+0000 7f67397fa640 1 -- 172.21.15.3:0/2702382432 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f67300547c0 con 0x7f673c0b1b00 2024-03-25T18:34:16.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.571+0000 7f67397fa640 10 client.4594.objecter ms_dispatch 0x7f673c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:16.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.571+0000 7f67397fa640 3 client.4594.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:16.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.571+0000 7f67397fa640 3 client.4594.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:16.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.571+0000 7f67397fa640 20 client.4594.objecter dump_active .. 0 homeless 2024-03-25T18:34:16.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.575+0000 7f67397fa640 1 -- 172.21.15.3:0/2702382432 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6730050100 con 0x7f673c0b1b00 2024-03-25T18:34:16.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.575+0000 7f67397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:16.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.575+0000 7f67397fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:16.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.831+0000 7f6742fd4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:16.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.831+0000 7f6742fd4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:16.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.831+0000 7f6742fd4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:34:16.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.831+0000 7f6742fd4640 1 -- 172.21.15.3:0/2702382432 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f673c0aa090 con 0x7f673c0b1b00 2024-03-25T18:34:16.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.850+0000 7f67397fa640 1 -- 172.21.15.3:0/2702382432 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f6730052200 con 0x7f673c0b1b00 2024-03-25T18:34:16.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.850+0000 7f67397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:16.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.850+0000 7f67397fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:16.850 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:34:16.850 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:16.854 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.853+0000 7f6742fd4640 10 librados: watch_flush enter 2024-03-25T18:34:16.854 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.853+0000 7f6742fd4640 10 librados: watch_flush exit 2024-03-25T18:34:16.854 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.853+0000 7f6742fd4640 20 client.4594.objecter shutdown clearing up homeless session... 2024-03-25T18:34:16.854 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.853+0000 7f6742fd4640 10 client.4594.objecter successfully canceled tick 2024-03-25T18:34:16.854 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.853+0000 7f6742fd4640 1 -- 172.21.15.3:0/2702382432 >> v1:172.21.15.3:6800/4195140966 conn(0x7f672006dd00 legacy=0x7f67200701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:16.854 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.853+0000 7f6742fd4640 10 monclient: shutdown 2024-03-25T18:34:16.854 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.853+0000 7f6742fd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:16.854 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.853+0000 7f6742fd4640 1 -- 172.21.15.3:0/2702382432 >> v1:172.21.15.16:6789/0 conn(0x7f673c0b1b00 legacy=0x7f673c133e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:16.854 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.853+0000 7f6742fd4640 1 -- 172.21.15.3:0/2702382432 shutdown_connections 2024-03-25T18:34:16.854 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.853+0000 7f6742fd4640 1 -- 172.21.15.3:0/2702382432 >> 172.21.15.3:0/2702382432 conn(0x7f673c0617a0 msgr2=0x7f673c0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:16.854 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.853+0000 7f6742fd4640 1 -- 172.21.15.3:0/2702382432 shutdown_connections 2024-03-25T18:34:16.855 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.853+0000 7f6742fd4640 1 -- 172.21.15.3:0/2702382432 wait complete. 2024-03-25T18:34:16.855 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:16.853+0000 7f6742fd4640 1 librados: shutdown 2024-03-25T18:34:16.865 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:23.198 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:09:36.927 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:23.205 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:09:36.934 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:23.244 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:09:36.973 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:23.266 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:09:36.995 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:23.272 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:09:36.964 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:23.276 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:09:37.005 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:23.277 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:09:36.969 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:23.307 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:09:37.000 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:23.309 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:09:37.038 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:23.671 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:23.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.785+0000 7f14ef765640 10 monclient: get_monmap_and_config 2024-03-25T18:34:23.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.785+0000 7f14ef765640 10 monclient: build_initial_monmap 2024-03-25T18:34:23.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.785+0000 7f14ef765640 10 monclient: monmap: 2024-03-25T18:34:23.786 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:34:23.786 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:23.786 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:34:23.786338+0000 2024-03-25T18:34:23.786 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:34:23.786338+0000 2024-03-25T18:34:23.786 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:23.786 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:23.787 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:34:23.787 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:34:23.787 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:34:23.787 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:23.788 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.785+0000 7f14ef765640 1 Processor -- start 2024-03-25T18:34:23.788 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.786+0000 7f14ef765640 1 -- start start 2024-03-25T18:34:23.788 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.786+0000 7f14ef765640 10 monclient: init 2024-03-25T18:34:23.788 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.787+0000 7f14ef765640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:23.788 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.787+0000 7f14ef765640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:34:23.788 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.787+0000 7f14ef765640 10 monclient(hunting): picked mon.noname-b con 0x7f14e80b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:34:23.788 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.787+0000 7f14ef765640 10 monclient(hunting): picked mon.noname-c con 0x7f14e80b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:34:23.789 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.787+0000 7f14ef765640 10 monclient(hunting): picked mon.noname-a con 0x7f14e813c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:34:23.789 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.787+0000 7f14ef765640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e80ab0e0 con 0x7f14e813c480 2024-03-25T18:34:23.789 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.787+0000 7f14ef765640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e80ab3c0 con 0x7f14e80b4670 2024-03-25T18:34:23.789 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.787+0000 7f14ef765640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e80a9f30 con 0x7f14e80b4ff0 2024-03-25T18:34:23.789 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.787+0000 7f14ef765640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:23.789 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.787+0000 7f14ef765640 10 monclient(hunting): authenticate will time out at 7918.047686s 2024-03-25T18:34:23.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.799+0000 7f14ed501640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f14e80b4ff0 0x7f14e813bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:35782/0 (socket says 172.21.15.3:35782) 2024-03-25T18:34:23.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.799+0000 7f14ed501640 1 -- 172.21.15.3:0/327813037 learned_addr learned my addr 172.21.15.3:0/327813037 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:34:23.818 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.816+0000 7f14dffff640 1 -- 172.21.15.3:0/327813037 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957622796 0 0) 0x7f14e80a9f30 con 0x7f14e80b4ff0 2024-03-25T18:34:23.818 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.816+0000 7f14dffff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:23.818 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.816+0000 7f14dffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:23.818 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.816+0000 7f14dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:23.818 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.816+0000 7f14dffff640 10 monclient(hunting): my global_id is 4601 2024-03-25T18:34:23.818 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.816+0000 7f14dffff640 1 -- 172.21.15.3:0/327813037 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14d4003520 con 0x7f14e80b4ff0 2024-03-25T18:34:23.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.818+0000 7f14dffff640 1 -- 172.21.15.3:0/327813037 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1029827995 0 0) 0x7f14e80ab3c0 con 0x7f14e80b4670 2024-03-25T18:34:23.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.818+0000 7f14dffff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:23.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.818+0000 7f14dffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:23.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.818+0000 7f14dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:23.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.818+0000 7f14dffff640 10 monclient(hunting): my global_id is 4597 2024-03-25T18:34:23.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.818+0000 7f14dffff640 1 -- 172.21.15.3:0/327813037 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14e80a9f30 con 0x7f14e80b4670 2024-03-25T18:34:23.824 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.823+0000 7f14dffff640 1 -- 172.21.15.3:0/327813037 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 954944510 0 0) 0x7f14e80a9f30 con 0x7f14e80b4670 2024-03-25T18:34:23.824 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.823+0000 7f14dffff640 1 -- 172.21.15.3:0/327813037 >> v1:172.21.15.3:6790/0 conn(0x7f14e80b4ff0 legacy=0x7f14e813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:23.824 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.823+0000 7f14dffff640 1 -- 172.21.15.3:0/327813037 >> v1:172.21.15.3:6789/0 conn(0x7f14e813c480 legacy=0x7f14e814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:23.824 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.823+0000 7f14dffff640 10 monclient: _finish_hunting 0 2024-03-25T18:34:23.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.823+0000 7f14dffff640 1 monclient: found mon.noname-c 2024-03-25T18:34:23.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.823+0000 7f14dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:23.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.823+0000 7f14dffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:34:23.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.823+0000 7f14dffff640 1 -- 172.21.15.3:0/327813037 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e80a9b70 con 0x7f14e80b4670 2024-03-25T18:34:23.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.823+0000 7f14dffff640 10 monclient: _finish_auth 0 2024-03-25T18:34:23.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.823+0000 7f14dffff640 10 monclient: _check_auth_tickets 2024-03-25T18:34:23.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.823+0000 7f14dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:23.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.823+0000 7f14dffff640 1 -- 172.21.15.3:0/327813037 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f14d8001670 con 0x7f14e80b4670 2024-03-25T18:34:23.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.823+0000 7f14ef765640 5 monclient: authenticate success, global_id 4597 2024-03-25T18:34:23.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.823+0000 7f14ef765640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:23.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.823+0000 7f14dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:23.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.823+0000 7f14dffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:23.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.823+0000 7f14dffff640 10 monclient: dump: 2024-03-25T18:34:23.825 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:23.826 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:23.826 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:23.826 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:23.826 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:23.826 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:23.826 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:23.826 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:23.826 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:23.826 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:23.826 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.823+0000 7f14ef765640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:23.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.837+0000 7f14dffff640 1 -- 172.21.15.3:0/327813037 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14d80017d0 con 0x7f14e80b4670 2024-03-25T18:34:23.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.837+0000 7f14dffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:23.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.837+0000 7f14dffff640 1 -- 172.21.15.3:0/327813037 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f14d8001be0 con 0x7f14e80b4670 2024-03-25T18:34:23.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.837+0000 7f14ef765640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:23.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.837+0000 7f14ef765640 4 set_mon_vals no callback set 2024-03-25T18:34:23.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.837+0000 7f14dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:23.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.837+0000 7f14dffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:23.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.837+0000 7f14dffff640 10 monclient: dump: 2024-03-25T18:34:23.839 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:23.839 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:23.839 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:23.839 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:23.839 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:23.839 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:23.840 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:23.840 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:23.840 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:23.840 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:23.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.838+0000 7f14ef765640 10 monclient: shutdown 2024-03-25T18:34:23.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.838+0000 7f14ef765640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:23.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.838+0000 7f14ef765640 1 -- 172.21.15.3:0/327813037 >> v1:172.21.15.16:6789/0 conn(0x7f14e80b4670 legacy=0x7f14e81425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:23.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.838+0000 7f14ef765640 1 -- 172.21.15.3:0/327813037 shutdown_connections 2024-03-25T18:34:23.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.838+0000 7f14ef765640 1 -- 172.21.15.3:0/327813037 >> 172.21.15.3:0/327813037 conn(0x7f14e80617a0 msgr2=0x7f14e8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:23.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.838+0000 7f14ef765640 1 -- 172.21.15.3:0/327813037 shutdown_connections 2024-03-25T18:34:23.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.838+0000 7f14ef765640 1 -- 172.21.15.3:0/327813037 wait complete. 2024-03-25T18:34:23.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.839+0000 7f14ef765640 10 monclient: build_initial_monmap 2024-03-25T18:34:23.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.839+0000 7f14ef765640 10 monclient: monmap: 2024-03-25T18:34:23.841 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:34:23.842 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:23.842 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:34:23.842 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:34:23.842 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:23.842 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:34:23.842 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:34:23.842 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:34:23.842 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:34:23.842 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:23.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.839+0000 7f14ef765640 1 librados: starting msgr at 2024-03-25T18:34:23.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.839+0000 7f14ef765640 1 librados: starting objecter 2024-03-25T18:34:23.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.840+0000 7f14ef765640 1 Processor -- start 2024-03-25T18:34:23.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.840+0000 7f14ef765640 1 -- start start 2024-03-25T18:34:23.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.840+0000 7f14ef765640 1 librados: setting wanted keys 2024-03-25T18:34:23.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.840+0000 7f14ef765640 1 librados: calling monclient init 2024-03-25T18:34:23.843 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.840+0000 7f14ef765640 10 monclient: init 2024-03-25T18:34:23.843 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.840+0000 7f14ef765640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:23.843 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.840+0000 7f14ef765640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:34:23.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.840+0000 7f14ef765640 10 monclient(hunting): picked mon.noname-a con 0x7f14e80b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:34:23.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.841+0000 7f14ef765640 10 monclient(hunting): picked mon.noname-b con 0x7f14e80b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:34:23.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.841+0000 7f14ef765640 10 monclient(hunting): picked mon.noname-c con 0x7f14e813c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:34:23.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.841+0000 7f14ef765640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e8145af0 con 0x7f14e80b4670 2024-03-25T18:34:23.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.841+0000 7f14ef765640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e8130d60 con 0x7f14e80b4ff0 2024-03-25T18:34:23.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.841+0000 7f14ef765640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e8146690 con 0x7f14e813c480 2024-03-25T18:34:23.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.841+0000 7f14ef765640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:23.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.841+0000 7f14ef765640 10 monclient(hunting): authenticate will time out at 7918.101539s 2024-03-25T18:34:23.855 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.854+0000 7f14ecd00640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f14e80b4ff0 0x7f14e812ff40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:40298/0 (socket says 172.21.15.3:40298) 2024-03-25T18:34:23.855 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.854+0000 7f14ecd00640 1 -- 172.21.15.3:0/1680429236 learned_addr learned my addr 172.21.15.3:0/1680429236 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:34:23.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.855+0000 7f14ddffb640 10 client.?.objecter ms_handle_connect 0x7f14e813c480 2024-03-25T18:34:23.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.855+0000 7f14ddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:23.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.859+0000 7f14ddffb640 10 client.?.objecter ms_handle_connect 0x7f14e80b4ff0 2024-03-25T18:34:23.860 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.859+0000 7f14ddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:23.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.860+0000 7f14ddffb640 1 -- 172.21.15.3:0/1680429236 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 613011681 0 0) 0x7f14e8146690 con 0x7f14e813c480 2024-03-25T18:34:23.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.860+0000 7f14ddffb640 10 monclient(hunting): _negotiate 2024-03-25T18:34:23.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.860+0000 7f14ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:23.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.860+0000 7f14ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:23.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.860+0000 7f14ddffb640 10 monclient(hunting): my global_id is 4604 2024-03-25T18:34:23.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.860+0000 7f14ddffb640 1 -- 172.21.15.3:0/1680429236 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14cc004eb0 con 0x7f14e813c480 2024-03-25T18:34:23.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.861+0000 7f14ddffb640 10 client.?.objecter ms_handle_connect 0x7f14e80b4670 2024-03-25T18:34:23.862 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.861+0000 7f14ddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:23.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.869+0000 7f14ddffb640 1 -- 172.21.15.3:0/1680429236 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 159211736 0 0) 0x7f14e8145af0 con 0x7f14e80b4670 2024-03-25T18:34:23.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.869+0000 7f14ddffb640 10 monclient(hunting): _negotiate 2024-03-25T18:34:23.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.869+0000 7f14ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:23.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.869+0000 7f14ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:23.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.869+0000 7f14ddffb640 10 monclient(hunting): my global_id is 4653 2024-03-25T18:34:23.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.869+0000 7f14ddffb640 1 -- 172.21.15.3:0/1680429236 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14e8146690 con 0x7f14e80b4670 2024-03-25T18:34:23.881 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.880+0000 7f14ddffb640 1 -- 172.21.15.3:0/1680429236 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1471795039 0 0) 0x7f14e8130d60 con 0x7f14e80b4ff0 2024-03-25T18:34:23.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.880+0000 7f14ddffb640 10 monclient(hunting): _negotiate 2024-03-25T18:34:23.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.880+0000 7f14ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:23.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.880+0000 7f14ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:23.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.880+0000 7f14ddffb640 10 monclient(hunting): my global_id is 4600 2024-03-25T18:34:23.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.880+0000 7f14ddffb640 1 -- 172.21.15.3:0/1680429236 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14e8145af0 con 0x7f14e80b4ff0 2024-03-25T18:34:23.883 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.882+0000 7f14ddffb640 1 -- 172.21.15.3:0/1680429236 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2261585350 0 0) 0x7f14cc004eb0 con 0x7f14e813c480 2024-03-25T18:34:23.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.882+0000 7f14ddffb640 1 -- 172.21.15.3:0/1680429236 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14e8130d60 con 0x7f14e813c480 2024-03-25T18:34:23.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.882+0000 7f14ddffb640 1 -- 172.21.15.3:0/1680429236 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f14e0001b20 con 0x7f14e813c480 2024-03-25T18:34:23.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.882+0000 7f14ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:23.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.882+0000 7f14ddffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:23.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.882+0000 7f14ddffb640 10 monclient(hunting): dump: 2024-03-25T18:34:23.884 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:23.884 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:23.884 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:23.884 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:23.884 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:23.884 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:23.884 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:23.884 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:23.884 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:23.885 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:23.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.882+0000 7f14ddffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:23.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.910+0000 7f14ddffb640 1 -- 172.21.15.3:0/1680429236 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3203004865 0 0) 0x7f14e8130d60 con 0x7f14e813c480 2024-03-25T18:34:23.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.910+0000 7f14ddffb640 1 -- 172.21.15.3:0/1680429236 >> v1:172.21.15.16:6789/0 conn(0x7f14e80b4ff0 legacy=0x7f14e812ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:23.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.910+0000 7f14ddffb640 1 -- 172.21.15.3:0/1680429236 >> v1:172.21.15.3:6789/0 conn(0x7f14e80b4670 legacy=0x7f14e8145f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:23.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.910+0000 7f14ddffb640 10 monclient: _finish_hunting 0 2024-03-25T18:34:23.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.910+0000 7f14ddffb640 1 monclient: found mon.c 2024-03-25T18:34:23.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.910+0000 7f14ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:23.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.910+0000 7f14ddffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:34:23.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.910+0000 7f14ddffb640 1 -- 172.21.15.3:0/1680429236 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e81456d0 con 0x7f14e813c480 2024-03-25T18:34:23.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.910+0000 7f14ddffb640 10 monclient: _finish_auth 0 2024-03-25T18:34:23.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.910+0000 7f14ddffb640 10 monclient: _check_auth_tickets 2024-03-25T18:34:23.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.910+0000 7f14ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:23.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.910+0000 7f14ef765640 5 monclient: authenticate success, global_id 4604 2024-03-25T18:34:23.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.910+0000 7f14ef765640 10 monclient: _renew_subs 2024-03-25T18:34:23.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.910+0000 7f14ef765640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:34:23.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.910+0000 7f14ef765640 1 -- 172.21.15.3:0/1680429236 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e8143fe0 con 0x7f14e813c480 2024-03-25T18:34:23.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.910+0000 7f14ef765640 10 client.4604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:23.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.910+0000 7f14ef765640 10 monclient: _renew_subs 2024-03-25T18:34:23.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.910+0000 7f14ef765640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:34:23.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.910+0000 7f14ef765640 1 -- 172.21.15.3:0/1680429236 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e813cb00 con 0x7f14e813c480 2024-03-25T18:34:23.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.910+0000 7f14ef765640 1 librados: init done 2024-03-25T18:34:23.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.911+0000 7f14ef765640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:23.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.911+0000 7f14ef765640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:23.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.911+0000 7f14ef765640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:34:23.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:23.911+0000 7f14ef765640 1 -- 172.21.15.3:0/1680429236 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14e8145410 con 0x7f14e813c480 2024-03-25T18:34:24.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.241+0000 7f14ddffb640 1 -- 172.21.15.3:0/1680429236 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14e0002030 con 0x7f14e813c480 2024-03-25T18:34:24.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.241+0000 7f14ddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:24.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.241+0000 7f14ddffb640 1 -- 172.21.15.3:0/1680429236 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f14e0003970 con 0x7f14e813c480 2024-03-25T18:34:24.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.241+0000 7f14ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:24.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.241+0000 7f14ddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:24.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.241+0000 7f14ddffb640 10 monclient: dump: 2024-03-25T18:34:24.244 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:24.244 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:24.244 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:24.244 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:24.244 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:24.244 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:24.244 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:24.244 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:24.244 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:24.244 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:24.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.241+0000 7f14df7fe640 4 set_mon_vals no callback set 2024-03-25T18:34:24.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.242+0000 7f14ddffb640 1 -- 172.21.15.3:0/1680429236 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f14e0003b90 con 0x7f14e813c480 2024-03-25T18:34:24.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.243+0000 7f14ed501640 10 monclient: get_auth_request con 0x7f14cc06dda0 auth_method 0 2024-03-25T18:34:24.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.243+0000 7f14ddffb640 10 client.4604.objecter ms_handle_connect 0x7f14cc06dda0 2024-03-25T18:34:24.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.249+0000 7f14ddffb640 1 -- 172.21.15.3:0/1680429236 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f14e00536e0 con 0x7f14e813c480 2024-03-25T18:34:24.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.249+0000 7f14ddffb640 10 client.4604.objecter ms_dispatch 0x7f14e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:24.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.249+0000 7f14ddffb640 3 client.4604.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:24.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.249+0000 7f14ddffb640 3 client.4604.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:24.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.249+0000 7f14ddffb640 20 client.4604.objecter dump_active .. 0 homeless 2024-03-25T18:34:24.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.249+0000 7f14ddffb640 1 -- 172.21.15.3:0/1680429236 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f14e004f020 con 0x7f14e813c480 2024-03-25T18:34:24.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.249+0000 7f14ddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:24.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.249+0000 7f14ddffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:24.466 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.465+0000 7f14ef765640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:24.466 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.465+0000 7f14ef765640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:24.466 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.465+0000 7f14ef765640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:34:24.467 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.465+0000 7f14ef765640 1 -- 172.21.15.3:0/1680429236 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14e8145ca0 con 0x7f14e813c480 2024-03-25T18:34:24.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.517+0000 7f14ddffb640 1 -- 172.21.15.3:0/1680429236 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f14e8145ca0 con 0x7f14e813c480 2024-03-25T18:34:24.519 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.517+0000 7f14ddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:24.519 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.517+0000 7f14ddffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:24.519 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:34:24.519 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:24.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.520+0000 7f14ef765640 10 librados: watch_flush enter 2024-03-25T18:34:24.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.520+0000 7f14ef765640 10 librados: watch_flush exit 2024-03-25T18:34:24.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.520+0000 7f14ef765640 20 client.4604.objecter shutdown clearing up homeless session... 2024-03-25T18:34:24.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.520+0000 7f14ef765640 10 client.4604.objecter successfully canceled tick 2024-03-25T18:34:24.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.520+0000 7f14ef765640 1 -- 172.21.15.3:0/1680429236 >> v1:172.21.15.3:6800/4195140966 conn(0x7f14cc06dda0 legacy=0x7f14cc070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:24.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.520+0000 7f14ef765640 10 monclient: shutdown 2024-03-25T18:34:24.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.520+0000 7f14ef765640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:24.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.520+0000 7f14ef765640 1 -- 172.21.15.3:0/1680429236 >> v1:172.21.15.3:6790/0 conn(0x7f14e813c480 legacy=0x7f14e8130650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:24.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.520+0000 7f14ef765640 1 -- 172.21.15.3:0/1680429236 shutdown_connections 2024-03-25T18:34:24.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.520+0000 7f14ef765640 1 -- 172.21.15.3:0/1680429236 >> 172.21.15.3:0/1680429236 conn(0x7f14e80617a0 msgr2=0x7f14e8141ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:24.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.520+0000 7f14ef765640 1 -- 172.21.15.3:0/1680429236 shutdown_connections 2024-03-25T18:34:24.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.520+0000 7f14ef765640 1 -- 172.21.15.3:0/1680429236 wait complete. 2024-03-25T18:34:24.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:24.520+0000 7f14ef765640 1 librados: shutdown 2024-03-25T18:34:24.533 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:31.340 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:31.459 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.458+0000 7fe60d196640 10 monclient: get_monmap_and_config 2024-03-25T18:34:31.459 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.458+0000 7fe60d196640 10 monclient: build_initial_monmap 2024-03-25T18:34:31.460 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.458+0000 7fe60d196640 10 monclient: monmap: 2024-03-25T18:34:31.460 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:34:31.460 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:31.460 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:34:31.459604+0000 2024-03-25T18:34:31.460 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:34:31.459604+0000 2024-03-25T18:34:31.460 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:31.460 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:31.460 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:34:31.460 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:34:31.460 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:34:31.460 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:31.462 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.459+0000 7fe60d196640 1 Processor -- start 2024-03-25T18:34:31.462 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.459+0000 7fe60d196640 1 -- start start 2024-03-25T18:34:31.462 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.459+0000 7fe60d196640 10 monclient: init 2024-03-25T18:34:31.462 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.459+0000 7fe60d196640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:31.462 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.459+0000 7fe60d196640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:34:31.462 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.459+0000 7fe60d196640 10 monclient(hunting): picked mon.noname-b con 0x7fe6080b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:34:31.462 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.459+0000 7fe60d196640 10 monclient(hunting): picked mon.noname-c con 0x7fe6080b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:34:31.462 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.459+0000 7fe60d196640 10 monclient(hunting): picked mon.noname-a con 0x7fe60813c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:34:31.463 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.459+0000 7fe60d196640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6080ab0e0 con 0x7fe60813c480 2024-03-25T18:34:31.463 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.459+0000 7fe60d196640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6080ab3c0 con 0x7fe6080b4670 2024-03-25T18:34:31.463 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.459+0000 7fe60d196640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6080a9f30 con 0x7fe6080b4ff0 2024-03-25T18:34:31.463 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.459+0000 7fe60d196640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:31.463 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.459+0000 7fe60d196640 10 monclient(hunting): authenticate will time out at 7925.720259s 2024-03-25T18:34:31.463 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.462+0000 7fe60659c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe6080b4670 0x7fe6081425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:52928/0 (socket says 172.21.15.3:52928) 2024-03-25T18:34:31.463 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.462+0000 7fe60659c640 1 -- 172.21.15.3:0/3648477717 learned_addr learned my addr 172.21.15.3:0/3648477717 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:34:31.479 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.478+0000 7fe605d9b640 1 -- 172.21.15.3:0/3648477717 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4114647722 0 0) 0x7fe6080a9f30 con 0x7fe6080b4ff0 2024-03-25T18:34:31.480 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.478+0000 7fe605d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:34:31.480 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.478+0000 7fe605d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:31.480 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.478+0000 7fe605d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:31.480 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.478+0000 7fe605d9b640 10 monclient(hunting): my global_id is 4607 2024-03-25T18:34:31.480 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.478+0000 7fe605d9b640 1 -- 172.21.15.3:0/3648477717 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5f0003520 con 0x7fe6080b4ff0 2024-03-25T18:34:31.480 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.478+0000 7fe605d9b640 1 -- 172.21.15.3:0/3648477717 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912597353 0 0) 0x7fe6080ab0e0 con 0x7fe60813c480 2024-03-25T18:34:31.480 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.478+0000 7fe605d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:34:31.480 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.478+0000 7fe605d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:31.480 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.478+0000 7fe605d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:31.480 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.478+0000 7fe605d9b640 10 monclient(hunting): my global_id is 4656 2024-03-25T18:34:31.480 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.478+0000 7fe605d9b640 1 -- 172.21.15.3:0/3648477717 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6080a9f30 con 0x7fe60813c480 2024-03-25T18:34:31.482 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.481+0000 7fe605d9b640 1 -- 172.21.15.3:0/3648477717 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3955835341 0 0) 0x7fe6080ab3c0 con 0x7fe6080b4670 2024-03-25T18:34:31.482 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.481+0000 7fe605d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:34:31.482 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.481+0000 7fe605d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:31.483 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.481+0000 7fe605d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:31.483 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.481+0000 7fe605d9b640 10 monclient(hunting): my global_id is 4603 2024-03-25T18:34:31.483 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.481+0000 7fe605d9b640 1 -- 172.21.15.3:0/3648477717 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6080ab0e0 con 0x7fe6080b4670 2024-03-25T18:34:31.486 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.484+0000 7fe605d9b640 1 -- 172.21.15.3:0/3648477717 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2335500337 0 0) 0x7fe5f0003520 con 0x7fe6080b4ff0 2024-03-25T18:34:31.486 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.484+0000 7fe605d9b640 1 -- 172.21.15.3:0/3648477717 >> v1:172.21.15.16:6789/0 conn(0x7fe6080b4670 legacy=0x7fe6081425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:31.486 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.484+0000 7fe605d9b640 1 -- 172.21.15.3:0/3648477717 >> v1:172.21.15.3:6789/0 conn(0x7fe60813c480 legacy=0x7fe60814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:31.486 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.484+0000 7fe605d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:34:31.486 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.484+0000 7fe605d9b640 1 monclient: found mon.noname-b 2024-03-25T18:34:31.486 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.484+0000 7fe605d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:31.486 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.484+0000 7fe605d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-25T18:34:31.486 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.484+0000 7fe605d9b640 1 -- 172.21.15.3:0/3648477717 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6080a9b70 con 0x7fe6080b4ff0 2024-03-25T18:34:31.486 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.484+0000 7fe605d9b640 10 monclient: _finish_auth 0 2024-03-25T18:34:31.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.484+0000 7fe605d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:34:31.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.484+0000 7fe605d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:31.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.484+0000 7fe605d9b640 1 -- 172.21.15.3:0/3648477717 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fe5f8001690 con 0x7fe6080b4ff0 2024-03-25T18:34:31.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.484+0000 7fe60d196640 5 monclient: authenticate success, global_id 4607 2024-03-25T18:34:31.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.484+0000 7fe60d196640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:31.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.484+0000 7fe605d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:31.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.484+0000 7fe605d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:31.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.484+0000 7fe605d9b640 10 monclient: dump: 2024-03-25T18:34:31.487 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:31.487 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:31.488 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:31.488 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:31.488 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:31.488 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:31.488 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:31.488 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:31.488 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:31.488 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:31.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.484+0000 7fe60d196640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:31.495 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.493+0000 7fe605d9b640 1 -- 172.21.15.3:0/3648477717 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5f80017f0 con 0x7fe6080b4ff0 2024-03-25T18:34:31.495 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.493+0000 7fe605d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:31.495 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.493+0000 7fe605d9b640 1 -- 172.21.15.3:0/3648477717 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fe5f8001c00 con 0x7fe6080b4ff0 2024-03-25T18:34:31.495 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.493+0000 7fe605d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:31.495 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.493+0000 7fe605d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:31.496 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.493+0000 7fe605d9b640 10 monclient: dump: 2024-03-25T18:34:31.496 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:31.496 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:31.496 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:31.496 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:31.496 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:31.496 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:31.496 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:31.496 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:31.496 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:31.496 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:31.497 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.493+0000 7fe60d196640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:31.497 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.493+0000 7fe60d196640 4 set_mon_vals no callback set 2024-03-25T18:34:31.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.494+0000 7fe60d196640 10 monclient: shutdown 2024-03-25T18:34:31.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.494+0000 7fe60d196640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:31.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.494+0000 7fe60d196640 1 -- 172.21.15.3:0/3648477717 >> v1:172.21.15.3:6790/0 conn(0x7fe6080b4ff0 legacy=0x7fe60813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:31.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.495+0000 7fe60d196640 1 -- 172.21.15.3:0/3648477717 shutdown_connections 2024-03-25T18:34:31.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.495+0000 7fe60d196640 1 -- 172.21.15.3:0/3648477717 >> 172.21.15.3:0/3648477717 conn(0x7fe6080617a0 msgr2=0x7fe608061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:31.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.495+0000 7fe60d196640 1 -- 172.21.15.3:0/3648477717 shutdown_connections 2024-03-25T18:34:31.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.495+0000 7fe60d196640 1 -- 172.21.15.3:0/3648477717 wait complete. 2024-03-25T18:34:31.500 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.496+0000 7fe60d196640 10 monclient: build_initial_monmap 2024-03-25T18:34:31.500 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.496+0000 7fe60d196640 10 monclient: monmap: 2024-03-25T18:34:31.500 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:34:31.500 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:31.500 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:34:31.500 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:34:31.500 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:31.500 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:34:31.500 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:34:31.500 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:34:31.500 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:34:31.501 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:31.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.496+0000 7fe60d196640 1 librados: starting msgr at 2024-03-25T18:34:31.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.496+0000 7fe60d196640 1 librados: starting objecter 2024-03-25T18:34:31.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.497+0000 7fe60d196640 1 Processor -- start 2024-03-25T18:34:31.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.497+0000 7fe60d196640 1 -- start start 2024-03-25T18:34:31.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.497+0000 7fe60d196640 1 librados: setting wanted keys 2024-03-25T18:34:31.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.497+0000 7fe60d196640 1 librados: calling monclient init 2024-03-25T18:34:31.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.497+0000 7fe60d196640 10 monclient: init 2024-03-25T18:34:31.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.497+0000 7fe60d196640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:31.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.497+0000 7fe60d196640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:34:31.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.497+0000 7fe60d196640 10 monclient(hunting): picked mon.noname-b con 0x7fe6080b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:34:31.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.497+0000 7fe60d196640 10 monclient(hunting): picked mon.noname-a con 0x7fe6080b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:34:31.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.497+0000 7fe60d196640 10 monclient(hunting): picked mon.noname-c con 0x7fe60813c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:34:31.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.497+0000 7fe60d196640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe60813d2f0 con 0x7fe6080b4ff0 2024-03-25T18:34:31.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.497+0000 7fe60d196640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe608143810 con 0x7fe6080b4670 2024-03-25T18:34:31.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.497+0000 7fe60d196640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe60813e190 con 0x7fe60813c480 2024-03-25T18:34:31.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.497+0000 7fe60d196640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:31.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.497+0000 7fe60d196640 10 monclient(hunting): authenticate will time out at 7925.758219s 2024-03-25T18:34:31.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.502+0000 7fe60659c640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fe6080b4ff0 0x7fe60813ef10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:39976/0 (socket says 172.21.15.3:39976) 2024-03-25T18:34:31.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.502+0000 7fe60659c640 1 -- 172.21.15.3:0/3882479486 learned_addr learned my addr 172.21.15.3:0/3882479486 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:34:31.505 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.504+0000 7fe5ef7fe640 10 client.?.objecter ms_handle_connect 0x7fe6080b4670 2024-03-25T18:34:31.505 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.504+0000 7fe5ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:31.512 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.511+0000 7fe5ef7fe640 10 client.?.objecter ms_handle_connect 0x7fe6080b4ff0 2024-03-25T18:34:31.512 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.511+0000 7fe5ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:31.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.514+0000 7fe5ef7fe640 1 -- 172.21.15.3:0/3882479486 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 231878039 0 0) 0x7fe608143810 con 0x7fe6080b4670 2024-03-25T18:34:31.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.514+0000 7fe5ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:31.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.514+0000 7fe5ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:31.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.514+0000 7fe5ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:31.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.514+0000 7fe5ef7fe640 10 monclient(hunting): my global_id is 4606 2024-03-25T18:34:31.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.514+0000 7fe5ef7fe640 1 -- 172.21.15.3:0/3882479486 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5f4004eb0 con 0x7fe6080b4670 2024-03-25T18:34:31.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.520+0000 7fe5ef7fe640 1 -- 172.21.15.3:0/3882479486 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 608391534 0 0) 0x7fe60813d2f0 con 0x7fe6080b4ff0 2024-03-25T18:34:31.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.520+0000 7fe5ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:31.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.520+0000 7fe5ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:31.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.520+0000 7fe5ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:31.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.520+0000 7fe5ef7fe640 10 monclient(hunting): my global_id is 4659 2024-03-25T18:34:31.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.520+0000 7fe5ef7fe640 1 -- 172.21.15.3:0/3882479486 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe608143810 con 0x7fe6080b4ff0 2024-03-25T18:34:31.526 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.525+0000 7fe5ef7fe640 10 client.?.objecter ms_handle_connect 0x7fe60813c480 2024-03-25T18:34:31.526 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.525+0000 7fe5ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:31.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.526+0000 7fe5ef7fe640 1 -- 172.21.15.3:0/3882479486 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1652258959 0 0) 0x7fe5f4004eb0 con 0x7fe6080b4670 2024-03-25T18:34:31.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.526+0000 7fe5ef7fe640 1 -- 172.21.15.3:0/3882479486 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe60813d2f0 con 0x7fe6080b4670 2024-03-25T18:34:31.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.526+0000 7fe5ef7fe640 1 -- 172.21.15.3:0/3882479486 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fe5f8001b00 con 0x7fe6080b4670 2024-03-25T18:34:31.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.526+0000 7fe5ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:31.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.526+0000 7fe5ef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:31.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.526+0000 7fe5ef7fe640 10 monclient(hunting): dump: 2024-03-25T18:34:31.528 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:31.528 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:31.528 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:31.529 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:31.529 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:31.529 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:31.529 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:31.529 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:31.529 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:31.529 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:31.529 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.526+0000 7fe5ef7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:31.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.531+0000 7fe5ef7fe640 1 -- 172.21.15.3:0/3882479486 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 612493793 0 0) 0x7fe60813e190 con 0x7fe60813c480 2024-03-25T18:34:31.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.531+0000 7fe5ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:31.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.531+0000 7fe5ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:31.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.531+0000 7fe5ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:31.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.531+0000 7fe5ef7fe640 10 monclient(hunting): my global_id is 4610 2024-03-25T18:34:31.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.531+0000 7fe5ef7fe640 1 -- 172.21.15.3:0/3882479486 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5f4004eb0 con 0x7fe60813c480 2024-03-25T18:34:31.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.532+0000 7fe5ef7fe640 1 -- 172.21.15.3:0/3882479486 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3030100060 0 0) 0x7fe608143810 con 0x7fe6080b4ff0 2024-03-25T18:34:31.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.532+0000 7fe5ef7fe640 1 -- 172.21.15.3:0/3882479486 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe60813e190 con 0x7fe6080b4ff0 2024-03-25T18:34:31.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.532+0000 7fe5ef7fe640 1 -- 172.21.15.3:0/3882479486 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fe5fc0018f0 con 0x7fe6080b4ff0 2024-03-25T18:34:31.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.532+0000 7fe5ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:31.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.532+0000 7fe5ef7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:31.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.532+0000 7fe5ef7fe640 10 monclient(hunting): dump: 2024-03-25T18:34:31.534 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:31.535 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:31.535 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:31.535 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:31.535 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:31.535 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:31.535 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:31.535 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:31.535 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:31.535 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:31.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.535+0000 7fe5ef7fe640 1 -- 172.21.15.3:0/3882479486 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1224345696 0 0) 0x7fe60813d2f0 con 0x7fe6080b4670 2024-03-25T18:34:31.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.535+0000 7fe5ef7fe640 1 -- 172.21.15.3:0/3882479486 >> v1:172.21.15.3:6790/0 conn(0x7fe60813c480 legacy=0x7fe6081e2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:31.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.535+0000 7fe5ef7fe640 1 -- 172.21.15.3:0/3882479486 >> v1:172.21.15.3:6789/0 conn(0x7fe6080b4ff0 legacy=0x7fe60813ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:31.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.535+0000 7fe5ef7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:34:31.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.535+0000 7fe5ef7fe640 1 monclient: found mon.b 2024-03-25T18:34:31.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.535+0000 7fe5ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:31.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.535+0000 7fe5ef7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:34:31.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.535+0000 7fe5ef7fe640 1 -- 172.21.15.3:0/3882479486 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe60813e470 con 0x7fe6080b4670 2024-03-25T18:34:31.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.535+0000 7fe5ef7fe640 10 monclient: _finish_auth 0 2024-03-25T18:34:31.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.535+0000 7fe5ef7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:34:31.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.535+0000 7fe5ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:31.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.535+0000 7fe60d196640 5 monclient: authenticate success, global_id 4606 2024-03-25T18:34:31.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.535+0000 7fe60d196640 10 monclient: _renew_subs 2024-03-25T18:34:31.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.535+0000 7fe60d196640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:34:31.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.535+0000 7fe60d196640 1 -- 172.21.15.3:0/3882479486 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe608145e80 con 0x7fe6080b4670 2024-03-25T18:34:31.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.535+0000 7fe60d196640 10 client.4606.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:31.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.535+0000 7fe60d196640 10 monclient: _renew_subs 2024-03-25T18:34:31.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.535+0000 7fe60d196640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:34:31.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.535+0000 7fe60d196640 1 -- 172.21.15.3:0/3882479486 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe60813f800 con 0x7fe6080b4670 2024-03-25T18:34:31.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.535+0000 7fe60d196640 1 librados: init done 2024-03-25T18:34:31.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.536+0000 7fe60d196640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:31.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.536+0000 7fe60d196640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:31.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.536+0000 7fe60d196640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:34:31.540 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.536+0000 7fe60d196640 1 -- 172.21.15.3:0/3882479486 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe60813e890 con 0x7fe6080b4670 2024-03-25T18:34:31.852 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.851+0000 7fe5ef7fe640 1 -- 172.21.15.3:0/3882479486 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5f8001220 con 0x7fe6080b4670 2024-03-25T18:34:31.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.851+0000 7fe5ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:31.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.851+0000 7fe5ef7fe640 1 -- 172.21.15.3:0/3882479486 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fe5f80038a0 con 0x7fe6080b4670 2024-03-25T18:34:31.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.851+0000 7fe5ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:31.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.851+0000 7fe5ef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:31.854 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.851+0000 7fe5ef7fe640 10 monclient: dump: 2024-03-25T18:34:31.854 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:31.854 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:31.854 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:31.854 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:31.854 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:31.854 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:31.854 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:31.854 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:31.855 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:31.855 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:31.855 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.851+0000 7fe60559a640 4 set_mon_vals no callback set 2024-03-25T18:34:31.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.852+0000 7fe5ef7fe640 1 -- 172.21.15.3:0/3882479486 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7fe5f8019870 con 0x7fe6080b4670 2024-03-25T18:34:31.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.852+0000 7fe60659c640 10 monclient: get_auth_request con 0x7fe5f406df50 auth_method 0 2024-03-25T18:34:31.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.853+0000 7fe5ef7fe640 10 client.4606.objecter ms_handle_connect 0x7fe5f406df50 2024-03-25T18:34:31.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.857+0000 7fe5ef7fe640 1 -- 172.21.15.3:0/3882479486 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7fe5f8053500 con 0x7fe6080b4670 2024-03-25T18:34:31.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.857+0000 7fe5ef7fe640 10 client.4606.objecter ms_dispatch 0x7fe608000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:31.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.857+0000 7fe5ef7fe640 3 client.4606.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:31.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.857+0000 7fe5ef7fe640 3 client.4606.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:31.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.857+0000 7fe5ef7fe640 20 client.4606.objecter dump_active .. 0 homeless 2024-03-25T18:34:31.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.858+0000 7fe5ef7fe640 1 -- 172.21.15.3:0/3882479486 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe5f804ee40 con 0x7fe6080b4670 2024-03-25T18:34:31.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.858+0000 7fe5ef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:31.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:31.858+0000 7fe5ef7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:32.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:32.083+0000 7fe60d196640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:32.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:32.083+0000 7fe60d196640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:32.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:32.083+0000 7fe60d196640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:34:32.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:32.083+0000 7fe60d196640 1 -- 172.21.15.3:0/3882479486 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe60806dbd0 con 0x7fe6080b4670 2024-03-25T18:34:32.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:32.217+0000 7fe5ef7fe640 1 -- 172.21.15.3:0/3882479486 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7fe5f8050f40 con 0x7fe6080b4670 2024-03-25T18:34:32.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:32.217+0000 7fe5ef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:32.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:32.217+0000 7fe5ef7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:32.219 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:34:32.219 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:32.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:32.219+0000 7fe60d196640 10 librados: watch_flush enter 2024-03-25T18:34:32.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:32.219+0000 7fe60d196640 10 librados: watch_flush exit 2024-03-25T18:34:32.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:32.219+0000 7fe60d196640 20 client.4606.objecter shutdown clearing up homeless session... 2024-03-25T18:34:32.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:32.219+0000 7fe60d196640 10 client.4606.objecter successfully canceled tick 2024-03-25T18:34:32.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:32.219+0000 7fe60d196640 1 -- 172.21.15.3:0/3882479486 >> v1:172.21.15.3:6800/4195140966 conn(0x7fe5f406df50 legacy=0x7fe5f4070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:32.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:32.219+0000 7fe60d196640 10 monclient: shutdown 2024-03-25T18:34:32.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:32.219+0000 7fe60d196640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:32.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:32.219+0000 7fe60d196640 1 -- 172.21.15.3:0/3882479486 >> v1:172.21.15.16:6789/0 conn(0x7fe6080b4670 legacy=0x7fe60813cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:32.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:32.219+0000 7fe60d196640 1 -- 172.21.15.3:0/3882479486 shutdown_connections 2024-03-25T18:34:32.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:32.219+0000 7fe60d196640 1 -- 172.21.15.3:0/3882479486 >> 172.21.15.3:0/3882479486 conn(0x7fe6080617a0 msgr2=0x7fe6081407a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:32.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:32.219+0000 7fe60d196640 1 -- 172.21.15.3:0/3882479486 shutdown_connections 2024-03-25T18:34:32.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:32.219+0000 7fe60d196640 1 -- 172.21.15.3:0/3882479486 wait complete. 2024-03-25T18:34:32.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:32.219+0000 7fe60d196640 1 librados: shutdown 2024-03-25T18:34:32.232 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:39.039 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:39.157 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.156+0000 7f6def304640 10 monclient: get_monmap_and_config 2024-03-25T18:34:39.157 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.156+0000 7f6def304640 10 monclient: build_initial_monmap 2024-03-25T18:34:39.157 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.156+0000 7f6def304640 10 monclient: monmap: 2024-03-25T18:34:39.157 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:34:39.157 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:39.157 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:34:39.157202+0000 2024-03-25T18:34:39.157 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:34:39.157202+0000 2024-03-25T18:34:39.157 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:39.157 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:39.158 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:34:39.158 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:34:39.158 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:34:39.158 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:39.159 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.157+0000 7f6def304640 1 Processor -- start 2024-03-25T18:34:39.159 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.157+0000 7f6def304640 1 -- start start 2024-03-25T18:34:39.159 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.157+0000 7f6def304640 10 monclient: init 2024-03-25T18:34:39.159 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.158+0000 7f6def304640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:39.160 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.158+0000 7f6def304640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:34:39.160 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.158+0000 7f6def304640 10 monclient(hunting): picked mon.noname-b con 0x7f6de806c3e0 addr v1:172.21.15.3:6790/0 2024-03-25T18:34:39.160 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.158+0000 7f6def304640 10 monclient(hunting): picked mon.noname-c con 0x7f6de80b0e00 addr v1:172.21.15.16:6789/0 2024-03-25T18:34:39.160 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.158+0000 7f6def304640 10 monclient(hunting): picked mon.noname-a con 0x7f6de81465b0 addr v1:172.21.15.3:6789/0 2024-03-25T18:34:39.160 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.158+0000 7f6def304640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de81469d0 con 0x7f6de81465b0 2024-03-25T18:34:39.160 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.158+0000 7f6def304640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de80adfc0 con 0x7f6de80b0e00 2024-03-25T18:34:39.160 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.158+0000 7f6def304640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de80ae2a0 con 0x7f6de806c3e0 2024-03-25T18:34:39.160 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.158+0000 7f6def304640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:39.160 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.158+0000 7f6def304640 10 monclient(hunting): authenticate will time out at 7933.418817s 2024-03-25T18:34:39.168 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.167+0000 7f6dec89f640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6de80b0e00 0x7f6de8145ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:60462/0 (socket says 172.21.15.3:60462) 2024-03-25T18:34:39.169 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.167+0000 7f6dec89f640 1 -- 172.21.15.3:0/435282236 learned_addr learned my addr 172.21.15.3:0/435282236 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:34:39.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.179+0000 7f6ddffff640 1 -- 172.21.15.3:0/435282236 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1347800515 0 0) 0x7f6de81469d0 con 0x7f6de81465b0 2024-03-25T18:34:39.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.179+0000 7f6ddffff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:39.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.179+0000 7f6ddffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:39.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.179+0000 7f6ddffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:39.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.179+0000 7f6ddffff640 10 monclient(hunting): my global_id is 4662 2024-03-25T18:34:39.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.179+0000 7f6ddffff640 1 -- 172.21.15.3:0/435282236 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dd4003520 con 0x7f6de81465b0 2024-03-25T18:34:39.186 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.185+0000 7f6ddffff640 1 -- 172.21.15.3:0/435282236 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950732419 0 0) 0x7f6de80adfc0 con 0x7f6de80b0e00 2024-03-25T18:34:39.186 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.185+0000 7f6ddffff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:39.186 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.185+0000 7f6ddffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:39.186 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.185+0000 7f6ddffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:39.186 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.185+0000 7f6ddffff640 10 monclient(hunting): my global_id is 4609 2024-03-25T18:34:39.186 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.185+0000 7f6ddffff640 1 -- 172.21.15.3:0/435282236 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6de81469d0 con 0x7f6de80b0e00 2024-03-25T18:34:39.189 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.187+0000 7f6ddffff640 1 -- 172.21.15.3:0/435282236 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 83274312 0 0) 0x7f6dd4003520 con 0x7f6de81465b0 2024-03-25T18:34:39.189 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.187+0000 7f6ddffff640 1 -- 172.21.15.3:0/435282236 >> v1:172.21.15.3:6790/0 conn(0x7f6de806c3e0 legacy=0x7f6de80b06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:39.189 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.187+0000 7f6ddffff640 1 -- 172.21.15.3:0/435282236 >> v1:172.21.15.16:6789/0 conn(0x7f6de80b0e00 legacy=0x7f6de8145ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:39.189 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.187+0000 7f6ddffff640 10 monclient: _finish_hunting 0 2024-03-25T18:34:39.189 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.187+0000 7f6ddffff640 1 monclient: found mon.noname-a 2024-03-25T18:34:39.189 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.187+0000 7f6ddffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:39.189 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.187+0000 7f6ddffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:34:39.189 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.187+0000 7f6ddffff640 1 -- 172.21.15.3:0/435282236 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de806d210 con 0x7f6de81465b0 2024-03-25T18:34:39.189 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.187+0000 7f6ddffff640 10 monclient: _finish_auth 0 2024-03-25T18:34:39.189 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.187+0000 7f6ddffff640 10 monclient: _check_auth_tickets 2024-03-25T18:34:39.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.187+0000 7f6ddffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:39.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.187+0000 7f6ddffff640 1 -- 172.21.15.3:0/435282236 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f6de0001a10 con 0x7f6de81465b0 2024-03-25T18:34:39.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.187+0000 7f6ddffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:39.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.187+0000 7f6ddffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:39.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.187+0000 7f6ddffff640 10 monclient: dump: 2024-03-25T18:34:39.190 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:39.190 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:39.190 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:39.190 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:39.190 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:39.190 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:39.190 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:39.190 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:39.190 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:39.190 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:39.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.187+0000 7f6def304640 5 monclient: authenticate success, global_id 4662 2024-03-25T18:34:39.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.187+0000 7f6def304640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:39.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.196+0000 7f6ddffff640 1 -- 172.21.15.3:0/435282236 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6de0001b70 con 0x7f6de81465b0 2024-03-25T18:34:39.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.196+0000 7f6ddffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:39.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.196+0000 7f6ddffff640 1 -- 172.21.15.3:0/435282236 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f6de0001f00 con 0x7f6de81465b0 2024-03-25T18:34:39.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.196+0000 7f6def304640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:39.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.196+0000 7f6def304640 4 set_mon_vals no callback set 2024-03-25T18:34:39.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.196+0000 7f6ddffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:39.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.196+0000 7f6ddffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:39.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.196+0000 7f6ddffff640 10 monclient: dump: 2024-03-25T18:34:39.198 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:39.198 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:39.198 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:39.198 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:39.198 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:39.198 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:39.199 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:39.199 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:39.199 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:39.199 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:39.202 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.197+0000 7f6def304640 10 monclient: shutdown 2024-03-25T18:34:39.202 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.197+0000 7f6def304640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:39.202 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.197+0000 7f6def304640 1 -- 172.21.15.3:0/435282236 >> v1:172.21.15.3:6789/0 conn(0x7f6de81465b0 legacy=0x7f6de814ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:39.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.197+0000 7f6def304640 1 -- 172.21.15.3:0/435282236 shutdown_connections 2024-03-25T18:34:39.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.197+0000 7f6def304640 1 -- 172.21.15.3:0/435282236 >> 172.21.15.3:0/435282236 conn(0x7f6de80b5cf0 msgr2=0x7f6de80b6100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:39.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.197+0000 7f6def304640 1 -- 172.21.15.3:0/435282236 shutdown_connections 2024-03-25T18:34:39.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.198+0000 7f6def304640 1 -- 172.21.15.3:0/435282236 wait complete. 2024-03-25T18:34:39.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.198+0000 7f6def304640 10 monclient: build_initial_monmap 2024-03-25T18:34:39.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.198+0000 7f6def304640 10 monclient: monmap: 2024-03-25T18:34:39.204 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:34:39.204 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:39.204 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:34:39.204 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:34:39.204 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:39.204 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:34:39.205 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:34:39.205 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:34:39.205 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:34:39.205 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:39.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.198+0000 7f6def304640 1 librados: starting msgr at 2024-03-25T18:34:39.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.198+0000 7f6def304640 1 librados: starting objecter 2024-03-25T18:34:39.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.199+0000 7f6def304640 1 Processor -- start 2024-03-25T18:34:39.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.199+0000 7f6def304640 1 -- start start 2024-03-25T18:34:39.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.199+0000 7f6def304640 1 librados: setting wanted keys 2024-03-25T18:34:39.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.199+0000 7f6def304640 1 librados: calling monclient init 2024-03-25T18:34:39.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.199+0000 7f6def304640 10 monclient: init 2024-03-25T18:34:39.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.199+0000 7f6def304640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:39.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.199+0000 7f6def304640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:34:39.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.199+0000 7f6def304640 10 monclient(hunting): picked mon.noname-b con 0x7f6de806c3e0 addr v1:172.21.15.16:6789/0 2024-03-25T18:34:39.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.199+0000 7f6def304640 10 monclient(hunting): picked mon.noname-c con 0x7f6de80b0e00 addr v1:172.21.15.3:6790/0 2024-03-25T18:34:39.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.199+0000 7f6def304640 10 monclient(hunting): picked mon.noname-a con 0x7f6de81465b0 addr v1:172.21.15.3:6789/0 2024-03-25T18:34:39.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.199+0000 7f6def304640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de81d5310 con 0x7f6de81465b0 2024-03-25T18:34:39.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.199+0000 7f6def304640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de8122440 con 0x7f6de806c3e0 2024-03-25T18:34:39.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.199+0000 7f6def304640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de8122fd0 con 0x7f6de80b0e00 2024-03-25T18:34:39.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.199+0000 7f6def304640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:39.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.199+0000 7f6def304640 10 monclient(hunting): authenticate will time out at 7933.460036s 2024-03-25T18:34:39.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.212+0000 7f6dec89f640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f6de80b0e00 0x7f6de81d7980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:50510/0 (socket says 172.21.15.3:50510) 2024-03-25T18:34:39.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.212+0000 7f6dec89f640 1 -- 172.21.15.3:0/3055760087 learned_addr learned my addr 172.21.15.3:0/3055760087 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:34:39.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.213+0000 7f6dddffb640 10 client.?.objecter ms_handle_connect 0x7f6de80b0e00 2024-03-25T18:34:39.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.213+0000 7f6dddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:39.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.215+0000 7f6dddffb640 10 client.?.objecter ms_handle_connect 0x7f6de81465b0 2024-03-25T18:34:39.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.215+0000 7f6dddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:39.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.216+0000 7f6dddffb640 10 client.?.objecter ms_handle_connect 0x7f6de806c3e0 2024-03-25T18:34:39.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.216+0000 7f6dddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:39.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.220+0000 7f6dddffb640 1 -- 172.21.15.3:0/3055760087 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3421994853 0 0) 0x7f6de81d5310 con 0x7f6de81465b0 2024-03-25T18:34:39.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.220+0000 7f6dddffb640 10 monclient(hunting): _negotiate 2024-03-25T18:34:39.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.220+0000 7f6dddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:39.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.220+0000 7f6dddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:39.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.220+0000 7f6dddffb640 10 monclient(hunting): my global_id is 4665 2024-03-25T18:34:39.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.220+0000 7f6dddffb640 1 -- 172.21.15.3:0/3055760087 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dcc004eb0 con 0x7f6de81465b0 2024-03-25T18:34:39.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.231+0000 7f6dddffb640 1 -- 172.21.15.3:0/3055760087 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3254222122 0 0) 0x7f6dcc004eb0 con 0x7f6de81465b0 2024-03-25T18:34:39.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.231+0000 7f6dddffb640 1 -- 172.21.15.3:0/3055760087 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6de81d5310 con 0x7f6de81465b0 2024-03-25T18:34:39.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.231+0000 7f6dddffb640 1 -- 172.21.15.3:0/3055760087 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f6de0001e80 con 0x7f6de81465b0 2024-03-25T18:34:39.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.231+0000 7f6dddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:39.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.231+0000 7f6dddffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:39.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.231+0000 7f6dddffb640 10 monclient(hunting): dump: 2024-03-25T18:34:39.232 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:39.232 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:39.233 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:39.233 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:39.233 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:39.233 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:39.233 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:39.233 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:39.233 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:39.233 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:39.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.231+0000 7f6dddffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:39.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.232+0000 7f6dddffb640 1 -- 172.21.15.3:0/3055760087 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4170630118 0 0) 0x7f6de8122fd0 con 0x7f6de80b0e00 2024-03-25T18:34:39.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.232+0000 7f6dddffb640 10 monclient(hunting): _negotiate 2024-03-25T18:34:39.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.232+0000 7f6dddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:39.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.232+0000 7f6dddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:39.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.232+0000 7f6dddffb640 10 monclient(hunting): my global_id is 4616 2024-03-25T18:34:39.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.232+0000 7f6dddffb640 1 -- 172.21.15.3:0/3055760087 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dcc004eb0 con 0x7f6de80b0e00 2024-03-25T18:34:39.237 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.236+0000 7f6dddffb640 1 -- 172.21.15.3:0/3055760087 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3856913403 0 0) 0x7f6de8122440 con 0x7f6de806c3e0 2024-03-25T18:34:39.237 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.236+0000 7f6dddffb640 10 monclient(hunting): _negotiate 2024-03-25T18:34:39.237 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.236+0000 7f6dddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:39.237 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.236+0000 7f6dddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:39.237 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.236+0000 7f6dddffb640 10 monclient(hunting): my global_id is 4612 2024-03-25T18:34:39.237 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.236+0000 7f6dddffb640 1 -- 172.21.15.3:0/3055760087 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6de8122fd0 con 0x7f6de806c3e0 2024-03-25T18:34:39.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.243+0000 7f6dddffb640 1 -- 172.21.15.3:0/3055760087 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2692965777 0 0) 0x7f6dcc004eb0 con 0x7f6de80b0e00 2024-03-25T18:34:39.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.243+0000 7f6dddffb640 1 -- 172.21.15.3:0/3055760087 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6de8122440 con 0x7f6de80b0e00 2024-03-25T18:34:39.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.243+0000 7f6dddffb640 1 -- 172.21.15.3:0/3055760087 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f6dd80019b0 con 0x7f6de80b0e00 2024-03-25T18:34:39.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.243+0000 7f6dddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:39.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.243+0000 7f6dddffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:39.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.243+0000 7f6dddffb640 10 monclient(hunting): dump: 2024-03-25T18:34:39.244 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:39.244 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:39.244 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:39.245 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:39.245 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:39.245 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:39.245 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:39.245 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:39.245 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:39.245 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:39.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.247+0000 7f6dddffb640 1 -- 172.21.15.3:0/3055760087 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 111444516 0 0) 0x7f6de81d5310 con 0x7f6de81465b0 2024-03-25T18:34:39.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.247+0000 7f6dddffb640 1 -- 172.21.15.3:0/3055760087 >> v1:172.21.15.3:6790/0 conn(0x7f6de80b0e00 legacy=0x7f6de81d7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:39.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.247+0000 7f6dddffb640 1 -- 172.21.15.3:0/3055760087 >> v1:172.21.15.16:6789/0 conn(0x7f6de806c3e0 legacy=0x7f6de813fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:39.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.247+0000 7f6dddffb640 10 monclient: _finish_hunting 0 2024-03-25T18:34:39.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.247+0000 7f6dddffb640 1 monclient: found mon.a 2024-03-25T18:34:39.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.247+0000 7f6dddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:39.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.247+0000 7f6dddffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:34:39.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.247+0000 7f6dddffb640 1 -- 172.21.15.3:0/3055760087 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de81232b0 con 0x7f6de81465b0 2024-03-25T18:34:39.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.247+0000 7f6dddffb640 10 monclient: _finish_auth 0 2024-03-25T18:34:39.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.247+0000 7f6dddffb640 10 monclient: _check_auth_tickets 2024-03-25T18:34:39.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.247+0000 7f6dddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:39.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.247+0000 7f6def304640 5 monclient: authenticate success, global_id 4665 2024-03-25T18:34:39.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.247+0000 7f6def304640 10 monclient: _renew_subs 2024-03-25T18:34:39.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.247+0000 7f6def304640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:34:39.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.247+0000 7f6def304640 1 -- 172.21.15.3:0/3055760087 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6de8120ec0 con 0x7f6de81465b0 2024-03-25T18:34:39.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.247+0000 7f6def304640 10 client.4665.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:39.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.247+0000 7f6def304640 10 monclient: _renew_subs 2024-03-25T18:34:39.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.247+0000 7f6def304640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:34:39.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.247+0000 7f6def304640 1 -- 172.21.15.3:0/3055760087 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6de8122180 con 0x7f6de81465b0 2024-03-25T18:34:39.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.247+0000 7f6def304640 1 librados: init done 2024-03-25T18:34:39.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.249+0000 7f6def304640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:39.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.249+0000 7f6def304640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:39.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.249+0000 7f6def304640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:34:39.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.249+0000 7f6def304640 1 -- 172.21.15.3:0/3055760087 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6de80b2450 con 0x7f6de81465b0 2024-03-25T18:34:39.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.254+0000 7f6dddffb640 1 -- 172.21.15.3:0/3055760087 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6de00020a0 con 0x7f6de81465b0 2024-03-25T18:34:39.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.254+0000 7f6dddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:39.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.254+0000 7f6ddf7fe640 4 set_mon_vals no callback set 2024-03-25T18:34:39.516 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.515+0000 7f6dddffb640 1 -- 172.21.15.3:0/3055760087 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f6de0003dc0 con 0x7f6de81465b0 2024-03-25T18:34:39.516 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.515+0000 7f6dddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:39.516 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.515+0000 7f6dddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:39.516 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.515+0000 7f6dddffb640 10 monclient: dump: 2024-03-25T18:34:39.516 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:39.516 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:39.516 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:39.516 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:39.516 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:39.516 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:39.516 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:39.516 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:39.516 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:39.516 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:39.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.559+0000 7f6dddffb640 1 -- 172.21.15.3:0/3055760087 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f6de0019d90 con 0x7f6de81465b0 2024-03-25T18:34:39.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.559+0000 7f6ded0a0640 10 monclient: get_auth_request con 0x7f6dcc06dff0 auth_method 0 2024-03-25T18:34:39.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.560+0000 7f6dddffb640 10 client.4665.objecter ms_handle_connect 0x7f6dcc06dff0 2024-03-25T18:34:39.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.567+0000 7f6dddffb640 1 -- 172.21.15.3:0/3055760087 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f6de00539e0 con 0x7f6de81465b0 2024-03-25T18:34:39.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.567+0000 7f6dddffb640 10 client.4665.objecter ms_dispatch 0x7f6de8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:39.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.567+0000 7f6dddffb640 3 client.4665.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:39.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.567+0000 7f6dddffb640 3 client.4665.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:39.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.567+0000 7f6dddffb640 20 client.4665.objecter dump_active .. 0 homeless 2024-03-25T18:34:39.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.567+0000 7f6dddffb640 1 -- 172.21.15.3:0/3055760087 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6de004f320 con 0x7f6de81465b0 2024-03-25T18:34:39.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.567+0000 7f6dddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:39.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.567+0000 7f6dddffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:39.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.830+0000 7f6def304640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:39.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.830+0000 7f6def304640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:39.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.830+0000 7f6def304640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:34:39.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.830+0000 7f6def304640 1 -- 172.21.15.3:0/3055760087 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6de804f730 con 0x7f6de81465b0 2024-03-25T18:34:39.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.852+0000 7f6dddffb640 1 -- 172.21.15.3:0/3055760087 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f6de0051420 con 0x7f6de81465b0 2024-03-25T18:34:39.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.852+0000 7f6dddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:39.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.852+0000 7f6dddffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:39.853 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:34:39.853 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:39.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.854+0000 7f6def304640 10 librados: watch_flush enter 2024-03-25T18:34:39.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.854+0000 7f6def304640 10 librados: watch_flush exit 2024-03-25T18:34:39.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.854+0000 7f6def304640 20 client.4665.objecter shutdown clearing up homeless session... 2024-03-25T18:34:39.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.854+0000 7f6def304640 10 client.4665.objecter successfully canceled tick 2024-03-25T18:34:39.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.854+0000 7f6def304640 1 -- 172.21.15.3:0/3055760087 >> v1:172.21.15.3:6800/4195140966 conn(0x7f6dcc06dff0 legacy=0x7f6dcc0704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:39.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.854+0000 7f6def304640 10 monclient: shutdown 2024-03-25T18:34:39.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.854+0000 7f6def304640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:39.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.854+0000 7f6def304640 1 -- 172.21.15.3:0/3055760087 >> v1:172.21.15.3:6789/0 conn(0x7f6de81465b0 legacy=0x7f6de81d8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:39.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.854+0000 7f6def304640 1 -- 172.21.15.3:0/3055760087 shutdown_connections 2024-03-25T18:34:39.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.854+0000 7f6def304640 1 -- 172.21.15.3:0/3055760087 >> 172.21.15.3:0/3055760087 conn(0x7f6de80b5cf0 msgr2=0x7f6de81454c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:39.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.854+0000 7f6def304640 1 -- 172.21.15.3:0/3055760087 shutdown_connections 2024-03-25T18:34:39.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.854+0000 7f6def304640 1 -- 172.21.15.3:0/3055760087 wait complete. 2024-03-25T18:34:39.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:39.854+0000 7f6def304640 1 librados: shutdown 2024-03-25T18:34:39.868 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:43.260 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:09:56.989 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:43.272 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:09:57.001 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:43.296 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:09:57.025 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:43.317 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:09:57.010 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:43.322 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:09:57.051 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:43.322 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:09:57.014 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:43.332 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:09:57.061 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:43.369 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:09:57.061 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:43.369 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:09:57.098 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:46.673 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:46.788 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.787+0000 7fe610fd0640 10 monclient: get_monmap_and_config 2024-03-25T18:34:46.789 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.787+0000 7fe610fd0640 10 monclient: build_initial_monmap 2024-03-25T18:34:46.789 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.787+0000 7fe610fd0640 10 monclient: monmap: 2024-03-25T18:34:46.789 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:34:46.789 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:46.789 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:34:46.788766+0000 2024-03-25T18:34:46.789 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:34:46.788766+0000 2024-03-25T18:34:46.789 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:46.789 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:46.789 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:34:46.789 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:34:46.789 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:34:46.789 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:46.790 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.788+0000 7fe610fd0640 1 Processor -- start 2024-03-25T18:34:46.790 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.788+0000 7fe610fd0640 1 -- start start 2024-03-25T18:34:46.791 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.788+0000 7fe610fd0640 10 monclient: init 2024-03-25T18:34:46.791 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.789+0000 7fe610fd0640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:46.791 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.789+0000 7fe610fd0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:34:46.791 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.789+0000 7fe610fd0640 10 monclient(hunting): picked mon.noname-a con 0x7fe60c0b4e70 addr v1:172.21.15.3:6789/0 2024-03-25T18:34:46.791 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.789+0000 7fe610fd0640 10 monclient(hunting): picked mon.noname-b con 0x7fe60c0b44b0 addr v1:172.21.15.3:6790/0 2024-03-25T18:34:46.791 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.789+0000 7fe610fd0640 10 monclient(hunting): picked mon.noname-c con 0x7fe60c0b1b00 addr v1:172.21.15.16:6789/0 2024-03-25T18:34:46.791 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.789+0000 7fe610fd0640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe60c0a96f0 con 0x7fe60c0b4e70 2024-03-25T18:34:46.791 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.789+0000 7fe610fd0640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe60c0ab0e0 con 0x7fe60c0b1b00 2024-03-25T18:34:46.791 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.789+0000 7fe610fd0640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe60c145f70 con 0x7fe60c0b44b0 2024-03-25T18:34:46.791 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.789+0000 7fe610fd0640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:46.791 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.789+0000 7fe610fd0640 10 monclient(hunting): authenticate will time out at 7941.049617s 2024-03-25T18:34:46.791 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.789+0000 7fe603fff640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fe60c0b44b0 0x7fe60c145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:55170/0 (socket says 172.21.15.3:55170) 2024-03-25T18:34:46.791 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.789+0000 7fe603fff640 1 -- 172.21.15.3:0/15860933 learned_addr learned my addr 172.21.15.3:0/15860933 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:34:46.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.885+0000 7fe609d9b640 1 -- 172.21.15.3:0/15860933 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 360418069 0 0) 0x7fe60c145f70 con 0x7fe60c0b44b0 2024-03-25T18:34:46.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.885+0000 7fe609d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:34:46.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.885+0000 7fe609d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:46.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.885+0000 7fe609d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:46.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.885+0000 7fe609d9b640 10 monclient(hunting): my global_id is 4619 2024-03-25T18:34:46.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.885+0000 7fe609d9b640 1 -- 172.21.15.3:0/15860933 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5f8003520 con 0x7fe60c0b44b0 2024-03-25T18:34:46.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.886+0000 7fe609d9b640 1 -- 172.21.15.3:0/15860933 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053345756 0 0) 0x7fe60c0ab0e0 con 0x7fe60c0b1b00 2024-03-25T18:34:46.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.886+0000 7fe609d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:34:46.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.886+0000 7fe609d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:46.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.886+0000 7fe609d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:46.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.886+0000 7fe609d9b640 10 monclient(hunting): my global_id is 4615 2024-03-25T18:34:46.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.886+0000 7fe609d9b640 1 -- 172.21.15.3:0/15860933 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe60c145f70 con 0x7fe60c0b1b00 2024-03-25T18:34:46.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.892+0000 7fe609d9b640 1 -- 172.21.15.3:0/15860933 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2084751130 0 0) 0x7fe60c0a96f0 con 0x7fe60c0b4e70 2024-03-25T18:34:46.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.892+0000 7fe609d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:34:46.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.892+0000 7fe609d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:46.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.892+0000 7fe609d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:46.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.892+0000 7fe609d9b640 10 monclient(hunting): my global_id is 4668 2024-03-25T18:34:46.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.892+0000 7fe609d9b640 1 -- 172.21.15.3:0/15860933 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe60c0ab0e0 con 0x7fe60c0b4e70 2024-03-25T18:34:46.896 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.894+0000 7fe609d9b640 1 -- 172.21.15.3:0/15860933 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 910794888 0 0) 0x7fe60c145f70 con 0x7fe60c0b1b00 2024-03-25T18:34:46.896 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.895+0000 7fe609d9b640 1 -- 172.21.15.3:0/15860933 >> v1:172.21.15.3:6790/0 conn(0x7fe60c0b44b0 legacy=0x7fe60c145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:46.896 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.895+0000 7fe609d9b640 1 -- 172.21.15.3:0/15860933 >> v1:172.21.15.3:6789/0 conn(0x7fe60c0b4e70 legacy=0x7fe60c13f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:46.896 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.895+0000 7fe609d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:34:46.896 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.895+0000 7fe609d9b640 1 monclient: found mon.noname-c 2024-03-25T18:34:46.896 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.895+0000 7fe609d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:46.896 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.895+0000 7fe609d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:34:46.896 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.895+0000 7fe609d9b640 1 -- 172.21.15.3:0/15860933 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe60c0a9b80 con 0x7fe60c0b1b00 2024-03-25T18:34:46.896 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.895+0000 7fe609d9b640 10 monclient: _finish_auth 0 2024-03-25T18:34:46.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.895+0000 7fe609d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:34:46.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.895+0000 7fe609d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:46.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.895+0000 7fe609d9b640 1 -- 172.21.15.3:0/15860933 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fe6040019f0 con 0x7fe60c0b1b00 2024-03-25T18:34:46.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.895+0000 7fe609d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:46.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.895+0000 7fe609d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:46.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.895+0000 7fe609d9b640 10 monclient: dump: 2024-03-25T18:34:46.897 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:46.897 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:46.897 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:46.897 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:46.897 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:46.897 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:46.897 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:46.897 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:46.897 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:46.898 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:46.898 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.895+0000 7fe610fd0640 5 monclient: authenticate success, global_id 4615 2024-03-25T18:34:46.898 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.895+0000 7fe610fd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:46.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.913+0000 7fe609d9b640 1 -- 172.21.15.3:0/15860933 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe604001b50 con 0x7fe60c0b1b00 2024-03-25T18:34:46.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.913+0000 7fe609d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:46.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.913+0000 7fe609d9b640 1 -- 172.21.15.3:0/15860933 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fe604001f60 con 0x7fe60c0b1b00 2024-03-25T18:34:46.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.913+0000 7fe609d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:46.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.913+0000 7fe609d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:46.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.913+0000 7fe609d9b640 10 monclient: dump: 2024-03-25T18:34:46.914 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:46.914 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:46.915 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:46.915 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:46.915 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:46.915 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:46.915 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:46.915 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:46.915 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:46.915 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:46.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.913+0000 7fe610fd0640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:46.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.913+0000 7fe610fd0640 4 set_mon_vals no callback set 2024-03-25T18:34:46.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.914+0000 7fe610fd0640 10 monclient: shutdown 2024-03-25T18:34:46.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.914+0000 7fe610fd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:46.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.914+0000 7fe610fd0640 1 -- 172.21.15.3:0/15860933 >> v1:172.21.15.16:6789/0 conn(0x7fe60c0b1b00 legacy=0x7fe60c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:46.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.914+0000 7fe610fd0640 1 -- 172.21.15.3:0/15860933 shutdown_connections 2024-03-25T18:34:46.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.914+0000 7fe610fd0640 1 -- 172.21.15.3:0/15860933 >> 172.21.15.3:0/15860933 conn(0x7fe60c0617a0 msgr2=0x7fe60c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:46.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.914+0000 7fe610fd0640 1 -- 172.21.15.3:0/15860933 shutdown_connections 2024-03-25T18:34:46.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.914+0000 7fe610fd0640 1 -- 172.21.15.3:0/15860933 wait complete. 2024-03-25T18:34:46.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.915+0000 7fe610fd0640 10 monclient: build_initial_monmap 2024-03-25T18:34:46.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.915+0000 7fe610fd0640 10 monclient: monmap: 2024-03-25T18:34:46.918 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:34:46.918 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:46.918 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:34:46.918 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:34:46.918 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:46.918 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:34:46.918 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:34:46.918 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:34:46.918 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:34:46.918 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:46.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.915+0000 7fe610fd0640 1 librados: starting msgr at 2024-03-25T18:34:46.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.915+0000 7fe610fd0640 1 librados: starting objecter 2024-03-25T18:34:46.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.915+0000 7fe610fd0640 1 Processor -- start 2024-03-25T18:34:46.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.915+0000 7fe610fd0640 1 -- start start 2024-03-25T18:34:46.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.915+0000 7fe610fd0640 1 librados: setting wanted keys 2024-03-25T18:34:46.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.915+0000 7fe610fd0640 1 librados: calling monclient init 2024-03-25T18:34:46.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.915+0000 7fe610fd0640 10 monclient: init 2024-03-25T18:34:46.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.916+0000 7fe610fd0640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:46.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.916+0000 7fe610fd0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:34:46.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.916+0000 7fe610fd0640 10 monclient(hunting): picked mon.noname-c con 0x7fe60c0b1b00 addr v1:172.21.15.3:6790/0 2024-03-25T18:34:46.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.916+0000 7fe610fd0640 10 monclient(hunting): picked mon.noname-b con 0x7fe60c0b44b0 addr v1:172.21.15.16:6789/0 2024-03-25T18:34:46.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.916+0000 7fe610fd0640 10 monclient(hunting): picked mon.noname-a con 0x7fe60c0b4e70 addr v1:172.21.15.3:6789/0 2024-03-25T18:34:46.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.916+0000 7fe610fd0640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe60c134950 con 0x7fe60c0b4e70 2024-03-25T18:34:46.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.916+0000 7fe610fd0640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe60c1369a0 con 0x7fe60c0b44b0 2024-03-25T18:34:46.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.916+0000 7fe610fd0640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe60c137560 con 0x7fe60c0b1b00 2024-03-25T18:34:46.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.916+0000 7fe610fd0640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:46.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.916+0000 7fe610fd0640 10 monclient(hunting): authenticate will time out at 7941.176639s 2024-03-25T18:34:46.924 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.923+0000 7fe603fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe60c0b44b0 0x7fe60c133b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:36800/0 (socket says 172.21.15.3:36800) 2024-03-25T18:34:46.924 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.923+0000 7fe603fff640 1 -- 172.21.15.3:0/1790412126 learned_addr learned my addr 172.21.15.3:0/1790412126 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:34:46.925 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.925+0000 7fe602ffd640 10 client.?.objecter ms_handle_connect 0x7fe60c0b44b0 2024-03-25T18:34:46.926 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.925+0000 7fe602ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:46.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.927+0000 7fe602ffd640 10 client.?.objecter ms_handle_connect 0x7fe60c0b1b00 2024-03-25T18:34:46.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.927+0000 7fe602ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:46.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.936+0000 7fe602ffd640 1 -- 172.21.15.3:0/1790412126 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2513524073 0 0) 0x7fe60c137560 con 0x7fe60c0b1b00 2024-03-25T18:34:46.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.936+0000 7fe602ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:46.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.936+0000 7fe602ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:46.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.936+0000 7fe602ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:46.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.936+0000 7fe602ffd640 10 monclient(hunting): my global_id is 4622 2024-03-25T18:34:46.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.936+0000 7fe602ffd640 1 -- 172.21.15.3:0/1790412126 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5ec004eb0 con 0x7fe60c0b1b00 2024-03-25T18:34:46.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.938+0000 7fe602ffd640 1 -- 172.21.15.3:0/1790412126 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 866948362 0 0) 0x7fe60c1369a0 con 0x7fe60c0b44b0 2024-03-25T18:34:46.940 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.938+0000 7fe602ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:46.940 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.938+0000 7fe602ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:46.940 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.938+0000 7fe602ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:46.940 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.938+0000 7fe602ffd640 10 monclient(hunting): my global_id is 4618 2024-03-25T18:34:46.940 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.938+0000 7fe602ffd640 1 -- 172.21.15.3:0/1790412126 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe60c137560 con 0x7fe60c0b44b0 2024-03-25T18:34:46.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.953+0000 7fe602ffd640 10 client.?.objecter ms_handle_connect 0x7fe60c0b4e70 2024-03-25T18:34:46.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.953+0000 7fe602ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:46.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.956+0000 7fe602ffd640 1 -- 172.21.15.3:0/1790412126 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 241880440 0 0) 0x7fe5ec004eb0 con 0x7fe60c0b1b00 2024-03-25T18:34:46.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.956+0000 7fe602ffd640 1 -- 172.21.15.3:0/1790412126 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe60c1369a0 con 0x7fe60c0b1b00 2024-03-25T18:34:46.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.956+0000 7fe602ffd640 1 -- 172.21.15.3:0/1790412126 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fe5fc001850 con 0x7fe60c0b1b00 2024-03-25T18:34:46.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.956+0000 7fe602ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:46.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.956+0000 7fe602ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:46.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.956+0000 7fe602ffd640 10 monclient(hunting): dump: 2024-03-25T18:34:46.958 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:46.958 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:46.958 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:46.958 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:46.958 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:46.958 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:46.958 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:46.958 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:46.959 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:46.959 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:46.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.956+0000 7fe602ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:46.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.962+0000 7fe602ffd640 1 -- 172.21.15.3:0/1790412126 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 861338285 0 0) 0x7fe60c137560 con 0x7fe60c0b44b0 2024-03-25T18:34:46.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.962+0000 7fe602ffd640 1 -- 172.21.15.3:0/1790412126 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5ec004eb0 con 0x7fe60c0b44b0 2024-03-25T18:34:46.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.962+0000 7fe602ffd640 1 -- 172.21.15.3:0/1790412126 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fe5f40018d0 con 0x7fe60c0b44b0 2024-03-25T18:34:46.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.962+0000 7fe602ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:46.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.962+0000 7fe602ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:46.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.962+0000 7fe602ffd640 10 monclient(hunting): dump: 2024-03-25T18:34:46.963 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:46.963 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:46.963 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:46.964 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:46.964 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:46.964 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:46.964 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:46.964 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:46.964 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:46.964 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:46.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.964+0000 7fe602ffd640 1 -- 172.21.15.3:0/1790412126 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3622678970 0 0) 0x7fe60c134950 con 0x7fe60c0b4e70 2024-03-25T18:34:46.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.964+0000 7fe602ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:46.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.964+0000 7fe602ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:46.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.964+0000 7fe602ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:46.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.964+0000 7fe602ffd640 10 monclient(hunting): my global_id is 4671 2024-03-25T18:34:46.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.964+0000 7fe602ffd640 1 -- 172.21.15.3:0/1790412126 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe60c137560 con 0x7fe60c0b4e70 2024-03-25T18:34:46.987 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.985+0000 7fe602ffd640 1 -- 172.21.15.3:0/1790412126 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3135206740 0 0) 0x7fe60c1369a0 con 0x7fe60c0b1b00 2024-03-25T18:34:46.987 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.985+0000 7fe602ffd640 1 -- 172.21.15.3:0/1790412126 >> v1:172.21.15.16:6789/0 conn(0x7fe60c0b44b0 legacy=0x7fe60c133b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:46.987 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.985+0000 7fe602ffd640 1 -- 172.21.15.3:0/1790412126 >> v1:172.21.15.3:6789/0 conn(0x7fe60c0b4e70 legacy=0x7fe60c134240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:46.987 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.985+0000 7fe602ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:34:46.987 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.985+0000 7fe602ffd640 1 monclient: found mon.c 2024-03-25T18:34:46.987 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.985+0000 7fe602ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:46.987 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.985+0000 7fe602ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:34:46.987 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.985+0000 7fe602ffd640 1 -- 172.21.15.3:0/1790412126 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe60c137840 con 0x7fe60c0b1b00 2024-03-25T18:34:46.987 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.985+0000 7fe602ffd640 10 monclient: _finish_auth 0 2024-03-25T18:34:46.987 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.985+0000 7fe602ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:34:46.987 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.985+0000 7fe602ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:46.987 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.985+0000 7fe610fd0640 5 monclient: authenticate success, global_id 4622 2024-03-25T18:34:46.987 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.985+0000 7fe610fd0640 10 monclient: _renew_subs 2024-03-25T18:34:46.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.985+0000 7fe610fd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:34:46.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.985+0000 7fe610fd0640 1 -- 172.21.15.3:0/1790412126 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe60c136480 con 0x7fe60c0b1b00 2024-03-25T18:34:46.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.985+0000 7fe610fd0640 10 client.4622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:46.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.985+0000 7fe610fd0640 10 monclient: _renew_subs 2024-03-25T18:34:46.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.985+0000 7fe610fd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:34:46.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.985+0000 7fe610fd0640 1 -- 172.21.15.3:0/1790412126 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe60c1360b0 con 0x7fe60c0b1b00 2024-03-25T18:34:46.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.985+0000 7fe610fd0640 1 librados: init done 2024-03-25T18:34:46.989 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.986+0000 7fe610fd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:46.989 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.986+0000 7fe610fd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:46.989 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.986+0000 7fe610fd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:34:46.989 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:46.986+0000 7fe610fd0640 1 -- 172.21.15.3:0/1790412126 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe60c04f730 con 0x7fe60c0b1b00 2024-03-25T18:34:47.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.317+0000 7fe602ffd640 1 -- 172.21.15.3:0/1790412126 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5fc001d60 con 0x7fe60c0b1b00 2024-03-25T18:34:47.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.317+0000 7fe602ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:47.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.317+0000 7fe602ffd640 1 -- 172.21.15.3:0/1790412126 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fe5fc0037f0 con 0x7fe60c0b1b00 2024-03-25T18:34:47.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.317+0000 7fe602ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:47.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.317+0000 7fe602ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:47.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.317+0000 7fe602ffd640 10 monclient: dump: 2024-03-25T18:34:47.319 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:47.319 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:47.319 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:47.319 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:47.319 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:47.319 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:47.319 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:47.319 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:47.320 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:47.320 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:47.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.317+0000 7fe60959a640 4 set_mon_vals no callback set 2024-03-25T18:34:47.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.318+0000 7fe602ffd640 1 -- 172.21.15.3:0/1790412126 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7fe5fc019840 con 0x7fe60c0b1b00 2024-03-25T18:34:47.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.319+0000 7fe603fff640 10 monclient: get_auth_request con 0x7fe5ec06df50 auth_method 0 2024-03-25T18:34:47.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.319+0000 7fe602ffd640 10 client.4622.objecter ms_handle_connect 0x7fe5ec06df50 2024-03-25T18:34:47.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.324+0000 7fe602ffd640 1 -- 172.21.15.3:0/1790412126 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7fe5fc0535e0 con 0x7fe60c0b1b00 2024-03-25T18:34:47.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.324+0000 7fe602ffd640 10 client.4622.objecter ms_dispatch 0x7fe60c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:47.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.324+0000 7fe602ffd640 3 client.4622.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:47.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.324+0000 7fe602ffd640 3 client.4622.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:47.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.324+0000 7fe602ffd640 20 client.4622.objecter dump_active .. 0 homeless 2024-03-25T18:34:47.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.324+0000 7fe602ffd640 1 -- 172.21.15.3:0/1790412126 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe5fc082e60 con 0x7fe60c0b1b00 2024-03-25T18:34:47.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.324+0000 7fe602ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:47.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.324+0000 7fe602ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:47.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.571+0000 7fe610fd0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:47.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.571+0000 7fe610fd0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:47.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.571+0000 7fe610fd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:34:47.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.571+0000 7fe610fd0640 1 -- 172.21.15.3:0/1790412126 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe60c0aa090 con 0x7fe60c0b1b00 2024-03-25T18:34:47.596 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.595+0000 7fe602ffd640 1 -- 172.21.15.3:0/1790412126 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7fe5fc04ef20 con 0x7fe60c0b1b00 2024-03-25T18:34:47.596 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.595+0000 7fe602ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:47.596 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.595+0000 7fe602ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:47.596 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:34:47.596 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:47.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.597+0000 7fe610fd0640 10 librados: watch_flush enter 2024-03-25T18:34:47.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.597+0000 7fe610fd0640 10 librados: watch_flush exit 2024-03-25T18:34:47.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.597+0000 7fe610fd0640 20 client.4622.objecter shutdown clearing up homeless session... 2024-03-25T18:34:47.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.597+0000 7fe610fd0640 10 client.4622.objecter successfully canceled tick 2024-03-25T18:34:47.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.597+0000 7fe610fd0640 1 -- 172.21.15.3:0/1790412126 >> v1:172.21.15.3:6800/4195140966 conn(0x7fe5ec06df50 legacy=0x7fe5ec070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:47.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.597+0000 7fe610fd0640 10 monclient: shutdown 2024-03-25T18:34:47.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.597+0000 7fe610fd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:47.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.597+0000 7fe610fd0640 1 -- 172.21.15.3:0/1790412126 >> v1:172.21.15.3:6790/0 conn(0x7fe60c0b1b00 legacy=0x7fe60c137fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:47.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.598+0000 7fe610fd0640 1 -- 172.21.15.3:0/1790412126 shutdown_connections 2024-03-25T18:34:47.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.598+0000 7fe610fd0640 1 -- 172.21.15.3:0/1790412126 >> 172.21.15.3:0/1790412126 conn(0x7fe60c0617a0 msgr2=0x7fe60c0b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:47.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.598+0000 7fe610fd0640 1 -- 172.21.15.3:0/1790412126 shutdown_connections 2024-03-25T18:34:47.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.598+0000 7fe610fd0640 1 -- 172.21.15.3:0/1790412126 wait complete. 2024-03-25T18:34:47.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:47.598+0000 7fe610fd0640 1 librados: shutdown 2024-03-25T18:34:47.610 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:54.416 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:54.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.534+0000 7faf89c98640 10 monclient: get_monmap_and_config 2024-03-25T18:34:54.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.534+0000 7faf89c98640 10 monclient: build_initial_monmap 2024-03-25T18:34:54.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.534+0000 7faf89c98640 10 monclient: monmap: 2024-03-25T18:34:54.535 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:34:54.535 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:54.535 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:34:54.535458+0000 2024-03-25T18:34:54.535 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:34:54.535458+0000 2024-03-25T18:34:54.536 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:54.536 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:54.536 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:34:54.536 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:34:54.536 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:34:54.536 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:54.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.535+0000 7faf89c98640 1 Processor -- start 2024-03-25T18:34:54.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.535+0000 7faf89c98640 1 -- start start 2024-03-25T18:34:54.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.535+0000 7faf89c98640 10 monclient: init 2024-03-25T18:34:54.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.535+0000 7faf89c98640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:54.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.535+0000 7faf89c98640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:34:54.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.535+0000 7faf89c98640 10 monclient(hunting): picked mon.noname-a con 0x7faf840b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:34:54.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.535+0000 7faf89c98640 10 monclient(hunting): picked mon.noname-c con 0x7faf840b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:34:54.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.535+0000 7faf89c98640 10 monclient(hunting): picked mon.noname-b con 0x7faf8413c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:34:54.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.535+0000 7faf89c98640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf840ab0e0 con 0x7faf840b4ff0 2024-03-25T18:34:54.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.535+0000 7faf89c98640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf840ab3c0 con 0x7faf840b4670 2024-03-25T18:34:54.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.536+0000 7faf89c98640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf840a9f30 con 0x7faf8413c480 2024-03-25T18:34:54.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.536+0000 7faf89c98640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:54.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.536+0000 7faf89c98640 10 monclient(hunting): authenticate will time out at 7948.796369s 2024-03-25T18:34:54.550 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.549+0000 7faf82ffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7faf840b4670 0x7faf841425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:32826/0 (socket says 172.21.15.3:32826) 2024-03-25T18:34:54.550 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.549+0000 7faf82ffd640 1 -- 172.21.15.3:0/1885400967 learned_addr learned my addr 172.21.15.3:0/1885400967 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:34:54.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.560+0000 7faf827fc640 1 -- 172.21.15.3:0/1885400967 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964947956 0 0) 0x7faf840a9f30 con 0x7faf8413c480 2024-03-25T18:34:54.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.560+0000 7faf827fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:54.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.560+0000 7faf827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:54.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.560+0000 7faf827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:54.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.560+0000 7faf827fc640 10 monclient(hunting): my global_id is 4625 2024-03-25T18:34:54.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.560+0000 7faf827fc640 1 -- 172.21.15.3:0/1885400967 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf6c003520 con 0x7faf8413c480 2024-03-25T18:34:54.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.564+0000 7faf827fc640 1 -- 172.21.15.3:0/1885400967 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3275353254 0 0) 0x7faf840ab3c0 con 0x7faf840b4670 2024-03-25T18:34:54.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.564+0000 7faf827fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:54.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.564+0000 7faf827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:54.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.564+0000 7faf827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:54.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.564+0000 7faf827fc640 10 monclient(hunting): my global_id is 4621 2024-03-25T18:34:54.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.564+0000 7faf827fc640 1 -- 172.21.15.3:0/1885400967 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf840a9f30 con 0x7faf840b4670 2024-03-25T18:34:54.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.572+0000 7faf827fc640 1 -- 172.21.15.3:0/1885400967 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2995613936 0 0) 0x7faf840a9f30 con 0x7faf840b4670 2024-03-25T18:34:54.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.572+0000 7faf827fc640 1 -- 172.21.15.3:0/1885400967 >> v1:172.21.15.3:6790/0 conn(0x7faf8413c480 legacy=0x7faf8414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:54.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.572+0000 7faf827fc640 1 -- 172.21.15.3:0/1885400967 >> v1:172.21.15.3:6789/0 conn(0x7faf840b4ff0 legacy=0x7faf8413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:54.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.572+0000 7faf827fc640 10 monclient: _finish_hunting 0 2024-03-25T18:34:54.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.572+0000 7faf827fc640 1 monclient: found mon.noname-c 2024-03-25T18:34:54.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.572+0000 7faf827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:54.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.572+0000 7faf827fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:34:54.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.572+0000 7faf827fc640 1 -- 172.21.15.3:0/1885400967 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf840a9b70 con 0x7faf840b4670 2024-03-25T18:34:54.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.572+0000 7faf827fc640 10 monclient: _finish_auth 0 2024-03-25T18:34:54.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.572+0000 7faf827fc640 10 monclient: _check_auth_tickets 2024-03-25T18:34:54.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.572+0000 7faf827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:54.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.572+0000 7faf827fc640 1 -- 172.21.15.3:0/1885400967 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7faf78001670 con 0x7faf840b4670 2024-03-25T18:34:54.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.572+0000 7faf89c98640 5 monclient: authenticate success, global_id 4621 2024-03-25T18:34:54.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.572+0000 7faf89c98640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:54.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.572+0000 7faf827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:54.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.573+0000 7faf827fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:54.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.573+0000 7faf827fc640 10 monclient: dump: 2024-03-25T18:34:54.575 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:54.575 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:54.575 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:54.575 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:54.575 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:54.575 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:54.575 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:54.575 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:54.575 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:54.576 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:54.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.573+0000 7faf89c98640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:54.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.598+0000 7faf827fc640 1 -- 172.21.15.3:0/1885400967 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf780017d0 con 0x7faf840b4670 2024-03-25T18:34:54.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.598+0000 7faf827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:54.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.598+0000 7faf827fc640 1 -- 172.21.15.3:0/1885400967 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7faf78001be0 con 0x7faf840b4670 2024-03-25T18:34:54.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.598+0000 7faf827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:54.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.598+0000 7faf827fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:54.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.598+0000 7faf827fc640 10 monclient: dump: 2024-03-25T18:34:54.600 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:54.600 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:54.600 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:54.600 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:54.600 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:54.600 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:54.600 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:54.600 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:54.600 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:54.600 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:54.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.598+0000 7faf89c98640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:54.600 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.598+0000 7faf89c98640 4 set_mon_vals no callback set 2024-03-25T18:34:54.602 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.599+0000 7faf89c98640 10 monclient: shutdown 2024-03-25T18:34:54.602 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.599+0000 7faf89c98640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:54.602 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.599+0000 7faf89c98640 1 -- 172.21.15.3:0/1885400967 >> v1:172.21.15.16:6789/0 conn(0x7faf840b4670 legacy=0x7faf841425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:54.602 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.599+0000 7faf89c98640 1 -- 172.21.15.3:0/1885400967 shutdown_connections 2024-03-25T18:34:54.602 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.599+0000 7faf89c98640 1 -- 172.21.15.3:0/1885400967 >> 172.21.15.3:0/1885400967 conn(0x7faf840617a0 msgr2=0x7faf84061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:54.602 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.599+0000 7faf89c98640 1 -- 172.21.15.3:0/1885400967 shutdown_connections 2024-03-25T18:34:54.602 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.599+0000 7faf89c98640 1 -- 172.21.15.3:0/1885400967 wait complete. 2024-03-25T18:34:54.602 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.600+0000 7faf89c98640 10 monclient: build_initial_monmap 2024-03-25T18:34:54.602 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.600+0000 7faf89c98640 10 monclient: monmap: 2024-03-25T18:34:54.602 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:34:54.602 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:54.602 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:34:54.602 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:34:54.603 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:54.603 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:34:54.603 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:34:54.603 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:34:54.603 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:34:54.603 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:54.603 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.600+0000 7faf89c98640 1 librados: starting msgr at 2024-03-25T18:34:54.603 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.600+0000 7faf89c98640 1 librados: starting objecter 2024-03-25T18:34:54.603 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.600+0000 7faf89c98640 1 Processor -- start 2024-03-25T18:34:54.603 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.601+0000 7faf89c98640 1 -- start start 2024-03-25T18:34:54.603 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.601+0000 7faf89c98640 1 librados: setting wanted keys 2024-03-25T18:34:54.603 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.601+0000 7faf89c98640 1 librados: calling monclient init 2024-03-25T18:34:54.603 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.601+0000 7faf89c98640 10 monclient: init 2024-03-25T18:34:54.604 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.601+0000 7faf89c98640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:54.604 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.601+0000 7faf89c98640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:34:54.604 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.601+0000 7faf89c98640 10 monclient(hunting): picked mon.noname-a con 0x7faf840b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:34:54.604 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.602+0000 7faf89c98640 10 monclient(hunting): picked mon.noname-b con 0x7faf840b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:34:54.605 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.602+0000 7faf89c98640 10 monclient(hunting): picked mon.noname-c con 0x7faf8413c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:34:54.605 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.602+0000 7faf89c98640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf8413c8f0 con 0x7faf840b4670 2024-03-25T18:34:54.605 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.602+0000 7faf89c98640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf8413c2b0 con 0x7faf840b4ff0 2024-03-25T18:34:54.605 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.602+0000 7faf89c98640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf84130ad0 con 0x7faf8413c480 2024-03-25T18:34:54.605 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.602+0000 7faf89c98640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:54.605 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.602+0000 7faf89c98640 10 monclient(hunting): authenticate will time out at 7948.862503s 2024-03-25T18:34:54.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.609+0000 7faf837fe640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7faf840b4670 0x7faf8413e1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53142/0 (socket says 172.21.15.3:53142) 2024-03-25T18:34:54.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.609+0000 7faf837fe640 1 -- 172.21.15.3:0/1772111379 learned_addr learned my addr 172.21.15.3:0/1772111379 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:34:54.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.610+0000 7faf6bfff640 10 client.?.objecter ms_handle_connect 0x7faf840b4670 2024-03-25T18:34:54.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.610+0000 7faf6bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:54.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.611+0000 7faf6bfff640 10 client.?.objecter ms_handle_connect 0x7faf840b4ff0 2024-03-25T18:34:54.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.611+0000 7faf6bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:54.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.613+0000 7faf6bfff640 10 client.?.objecter ms_handle_connect 0x7faf8413c480 2024-03-25T18:34:54.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.613+0000 7faf6bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:54.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.621+0000 7faf6bfff640 1 -- 172.21.15.3:0/1772111379 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 471263032 0 0) 0x7faf8413c8f0 con 0x7faf840b4670 2024-03-25T18:34:54.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.621+0000 7faf6bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:54.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.621+0000 7faf6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:54.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.621+0000 7faf6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:54.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.621+0000 7faf6bfff640 10 monclient(hunting): my global_id is 4677 2024-03-25T18:34:54.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.621+0000 7faf6bfff640 1 -- 172.21.15.3:0/1772111379 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf70004eb0 con 0x7faf840b4670 2024-03-25T18:34:54.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.632+0000 7faf6bfff640 1 -- 172.21.15.3:0/1772111379 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4035687965 0 0) 0x7faf8413c2b0 con 0x7faf840b4ff0 2024-03-25T18:34:54.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.632+0000 7faf6bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:54.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.632+0000 7faf6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:54.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.632+0000 7faf6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:54.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.632+0000 7faf6bfff640 10 monclient(hunting): my global_id is 4624 2024-03-25T18:34:54.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.632+0000 7faf6bfff640 1 -- 172.21.15.3:0/1772111379 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf8413c8f0 con 0x7faf840b4ff0 2024-03-25T18:34:54.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.632+0000 7faf6bfff640 1 -- 172.21.15.3:0/1772111379 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2260469521 0 0) 0x7faf84130ad0 con 0x7faf8413c480 2024-03-25T18:34:54.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.632+0000 7faf6bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:54.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.632+0000 7faf6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:54.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.632+0000 7faf6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:54.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.632+0000 7faf6bfff640 10 monclient(hunting): my global_id is 4628 2024-03-25T18:34:54.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.632+0000 7faf6bfff640 1 -- 172.21.15.3:0/1772111379 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf8413c2b0 con 0x7faf8413c480 2024-03-25T18:34:54.636 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.635+0000 7faf6bfff640 1 -- 172.21.15.3:0/1772111379 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3989250153 0 0) 0x7faf70004eb0 con 0x7faf840b4670 2024-03-25T18:34:54.636 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.635+0000 7faf6bfff640 1 -- 172.21.15.3:0/1772111379 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf84130ad0 con 0x7faf840b4670 2024-03-25T18:34:54.636 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.635+0000 7faf6bfff640 1 -- 172.21.15.3:0/1772111379 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7faf740019d0 con 0x7faf840b4670 2024-03-25T18:34:54.636 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.635+0000 7faf6bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:54.636 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.635+0000 7faf6bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:54.637 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.635+0000 7faf6bfff640 10 monclient(hunting): dump: 2024-03-25T18:34:54.637 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:54.637 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:54.637 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:54.637 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:54.637 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:54.637 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:54.637 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:54.637 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:54.637 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:54.637 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:54.637 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.635+0000 7faf6bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:54.649 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.648+0000 7faf6bfff640 1 -- 172.21.15.3:0/1772111379 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 159008502 0 0) 0x7faf8413c2b0 con 0x7faf8413c480 2024-03-25T18:34:54.649 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.648+0000 7faf6bfff640 1 -- 172.21.15.3:0/1772111379 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf70004eb0 con 0x7faf8413c480 2024-03-25T18:34:54.649 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.648+0000 7faf6bfff640 1 -- 172.21.15.3:0/1772111379 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7faf7c001c10 con 0x7faf8413c480 2024-03-25T18:34:54.649 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.648+0000 7faf6bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:54.649 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.648+0000 7faf6bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:54.649 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.648+0000 7faf6bfff640 10 monclient(hunting): dump: 2024-03-25T18:34:54.649 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:54.650 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:54.650 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:54.650 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:54.650 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:54.650 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:54.650 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:54.650 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:54.650 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:54.650 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:54.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.652+0000 7faf6bfff640 1 -- 172.21.15.3:0/1772111379 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3618323580 0 0) 0x7faf84130ad0 con 0x7faf840b4670 2024-03-25T18:34:54.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.652+0000 7faf6bfff640 1 -- 172.21.15.3:0/1772111379 >> v1:172.21.15.3:6790/0 conn(0x7faf8413c480 legacy=0x7faf841303c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:54.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.652+0000 7faf6bfff640 1 -- 172.21.15.3:0/1772111379 >> v1:172.21.15.16:6789/0 conn(0x7faf840b4ff0 legacy=0x7faf8412fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:54.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.652+0000 7faf6bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:34:54.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.652+0000 7faf6bfff640 1 monclient: found mon.a 2024-03-25T18:34:54.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.652+0000 7faf6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:54.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.652+0000 7faf6bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:34:54.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.652+0000 7faf6bfff640 1 -- 172.21.15.3:0/1772111379 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf8413e8f0 con 0x7faf840b4670 2024-03-25T18:34:54.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.652+0000 7faf6bfff640 10 monclient: _finish_auth 0 2024-03-25T18:34:54.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.652+0000 7faf6bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:34:54.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.652+0000 7faf6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:54.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.652+0000 7faf89c98640 5 monclient: authenticate success, global_id 4677 2024-03-25T18:34:54.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.652+0000 7faf89c98640 10 monclient: _renew_subs 2024-03-25T18:34:54.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.652+0000 7faf89c98640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:34:54.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.652+0000 7faf89c98640 1 -- 172.21.15.3:0/1772111379 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf8413d790 con 0x7faf840b4670 2024-03-25T18:34:54.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.652+0000 7faf89c98640 10 client.4677.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:54.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.652+0000 7faf89c98640 10 monclient: _renew_subs 2024-03-25T18:34:54.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.652+0000 7faf89c98640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:34:54.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.652+0000 7faf89c98640 1 -- 172.21.15.3:0/1772111379 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faf8413cb40 con 0x7faf840b4670 2024-03-25T18:34:54.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.652+0000 7faf89c98640 1 librados: init done 2024-03-25T18:34:54.656 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.653+0000 7faf89c98640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:54.656 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.653+0000 7faf89c98640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:54.656 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.653+0000 7faf89c98640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:34:54.656 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.653+0000 7faf89c98640 1 -- 172.21.15.3:0/1772111379 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf84146790 con 0x7faf840b4670 2024-03-25T18:34:54.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.976+0000 7faf6bfff640 1 -- 172.21.15.3:0/1772111379 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf74003610 con 0x7faf840b4670 2024-03-25T18:34:54.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.976+0000 7faf6bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:54.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.976+0000 7faf6bfff640 1 -- 172.21.15.3:0/1772111379 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7faf74003a20 con 0x7faf840b4670 2024-03-25T18:34:54.978 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.976+0000 7faf6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:54.978 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.976+0000 7faf6bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:54.978 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.976+0000 7faf6bfff640 10 monclient: dump: 2024-03-25T18:34:54.978 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:34:54.978 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:34:54.978 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:54.978 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:34:54.978 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:54.978 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:34:54.978 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:34:54.978 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:34:54.979 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:34:54.979 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:34:54.979 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.976+0000 7faf81ffb640 4 set_mon_vals no callback set 2024-03-25T18:34:54.979 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.977+0000 7faf6bfff640 1 -- 172.21.15.3:0/1772111379 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7faf74003c40 con 0x7faf840b4670 2024-03-25T18:34:54.979 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.977+0000 7faf82ffd640 10 monclient: get_auth_request con 0x7faf7006df70 auth_method 0 2024-03-25T18:34:54.979 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:54.978+0000 7faf6bfff640 10 client.4677.objecter ms_handle_connect 0x7faf7006df70 2024-03-25T18:34:55.001 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:55.000+0000 7faf6bfff640 1 -- 172.21.15.3:0/1772111379 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7faf74054860 con 0x7faf840b4670 2024-03-25T18:34:55.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:55.000+0000 7faf6bfff640 10 client.4677.objecter ms_dispatch 0x7faf84000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:55.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:55.000+0000 7faf6bfff640 3 client.4677.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:55.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:55.000+0000 7faf6bfff640 3 client.4677.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:55.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:55.000+0000 7faf6bfff640 20 client.4677.objecter dump_active .. 0 homeless 2024-03-25T18:34:55.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:55.000+0000 7faf6bfff640 1 -- 172.21.15.3:0/1772111379 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faf74050370 con 0x7faf840b4670 2024-03-25T18:34:55.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:55.000+0000 7faf6bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:55.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:55.000+0000 7faf6bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:55.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:55.217+0000 7faf89c98640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:55.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:55.217+0000 7faf89c98640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:55.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:55.217+0000 7faf89c98640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:34:55.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:55.217+0000 7faf89c98640 1 -- 172.21.15.3:0/1772111379 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf840688c0 con 0x7faf840b4670 2024-03-25T18:34:55.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:55.250+0000 7faf6bfff640 1 -- 172.21.15.3:0/1772111379 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7faf740522a0 con 0x7faf840b4670 2024-03-25T18:34:55.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:55.250+0000 7faf6bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:55.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:55.250+0000 7faf6bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:55.252 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:34:55.252 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:55.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:55.253+0000 7faf89c98640 10 librados: watch_flush enter 2024-03-25T18:34:55.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:55.253+0000 7faf89c98640 10 librados: watch_flush exit 2024-03-25T18:34:55.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:55.253+0000 7faf89c98640 20 client.4677.objecter shutdown clearing up homeless session... 2024-03-25T18:34:55.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:55.253+0000 7faf89c98640 10 client.4677.objecter successfully canceled tick 2024-03-25T18:34:55.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:55.253+0000 7faf89c98640 1 -- 172.21.15.3:0/1772111379 >> v1:172.21.15.3:6800/4195140966 conn(0x7faf7006df70 legacy=0x7faf70070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:55.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:55.253+0000 7faf89c98640 10 monclient: shutdown 2024-03-25T18:34:55.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:55.253+0000 7faf89c98640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:55.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:55.253+0000 7faf89c98640 1 -- 172.21.15.3:0/1772111379 >> v1:172.21.15.3:6789/0 conn(0x7faf840b4670 legacy=0x7faf8413e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:55.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:55.253+0000 7faf89c98640 1 -- 172.21.15.3:0/1772111379 shutdown_connections 2024-03-25T18:34:55.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:55.253+0000 7faf89c98640 1 -- 172.21.15.3:0/1772111379 >> 172.21.15.3:0/1772111379 conn(0x7faf840617a0 msgr2=0x7faf840b2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:55.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:55.253+0000 7faf89c98640 1 -- 172.21.15.3:0/1772111379 shutdown_connections 2024-03-25T18:34:55.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:55.253+0000 7faf89c98640 1 -- 172.21.15.3:0/1772111379 wait complete. 2024-03-25T18:34:55.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:34:55.253+0000 7faf89c98640 1 librados: shutdown 2024-03-25T18:34:55.267 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:02.074 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:02.167 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.166+0000 7f4043c50640 10 monclient: get_monmap_and_config 2024-03-25T18:35:02.168 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.166+0000 7f4043c50640 10 monclient: build_initial_monmap 2024-03-25T18:35:02.168 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.166+0000 7f4043c50640 10 monclient: monmap: 2024-03-25T18:35:02.168 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:35:02.168 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:02.168 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:35:02.167753+0000 2024-03-25T18:35:02.168 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:35:02.167753+0000 2024-03-25T18:35:02.168 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:02.168 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:02.168 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:35:02.168 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:35:02.169 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:35:02.169 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:02.170 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.167+0000 7f4043c50640 1 Processor -- start 2024-03-25T18:35:02.170 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.167+0000 7f4043c50640 1 -- start start 2024-03-25T18:35:02.170 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.167+0000 7f4043c50640 10 monclient: init 2024-03-25T18:35:02.171 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.168+0000 7f4043c50640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:02.171 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.168+0000 7f4043c50640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:35:02.171 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.168+0000 7f4043c50640 10 monclient(hunting): picked mon.noname-b con 0x7f403c0b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:35:02.171 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.168+0000 7f4043c50640 10 monclient(hunting): picked mon.noname-a con 0x7f403c0b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:35:02.171 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.168+0000 7f4043c50640 10 monclient(hunting): picked mon.noname-c con 0x7f403c13c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:35:02.171 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.168+0000 7f4043c50640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f403c0ab0e0 con 0x7f403c0b4670 2024-03-25T18:35:02.171 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.168+0000 7f4043c50640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f403c0ab3c0 con 0x7f403c13c480 2024-03-25T18:35:02.171 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.168+0000 7f4043c50640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f403c0a9f30 con 0x7f403c0b4ff0 2024-03-25T18:35:02.171 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.168+0000 7f4043c50640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:02.171 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.168+0000 7f4043c50640 10 monclient(hunting): authenticate will time out at 7956.428668s 2024-03-25T18:35:02.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.177+0000 7f40419ec640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f403c0b4ff0 0x7f403c13bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:53380/0 (socket says 172.21.15.3:53380) 2024-03-25T18:35:02.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.177+0000 7f40419ec640 1 -- 172.21.15.3:0/4048786672 learned_addr learned my addr 172.21.15.3:0/4048786672 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:35:02.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.195+0000 7f40409ea640 1 -- 172.21.15.3:0/4048786672 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2586606073 0 0) 0x7f403c0a9f30 con 0x7f403c0b4ff0 2024-03-25T18:35:02.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.195+0000 7f40409ea640 10 monclient(hunting): _negotiate 2024-03-25T18:35:02.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.195+0000 7f40409ea640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:02.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.195+0000 7f40409ea640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:02.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.195+0000 7f40409ea640 10 monclient(hunting): my global_id is 4631 2024-03-25T18:35:02.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.195+0000 7f40409ea640 1 -- 172.21.15.3:0/4048786672 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4028003520 con 0x7f403c0b4ff0 2024-03-25T18:35:02.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.197+0000 7f40409ea640 1 -- 172.21.15.3:0/4048786672 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 199356150 0 0) 0x7f403c0ab0e0 con 0x7f403c0b4670 2024-03-25T18:35:02.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.197+0000 7f40409ea640 10 monclient(hunting): _negotiate 2024-03-25T18:35:02.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.197+0000 7f40409ea640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:02.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.197+0000 7f40409ea640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:02.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.197+0000 7f40409ea640 10 monclient(hunting): my global_id is 4680 2024-03-25T18:35:02.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.197+0000 7f40409ea640 1 -- 172.21.15.3:0/4048786672 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f403c0a9f30 con 0x7f403c0b4670 2024-03-25T18:35:02.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.200+0000 7f40409ea640 1 -- 172.21.15.3:0/4048786672 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3891672990 0 0) 0x7f403c0ab3c0 con 0x7f403c13c480 2024-03-25T18:35:02.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.200+0000 7f40409ea640 10 monclient(hunting): _negotiate 2024-03-25T18:35:02.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.200+0000 7f40409ea640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:02.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.200+0000 7f40409ea640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:02.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.200+0000 7f40409ea640 10 monclient(hunting): my global_id is 4627 2024-03-25T18:35:02.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.200+0000 7f40409ea640 1 -- 172.21.15.3:0/4048786672 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f403c0ab0e0 con 0x7f403c13c480 2024-03-25T18:35:02.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.205+0000 7f40409ea640 1 -- 172.21.15.3:0/4048786672 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 926644075 0 0) 0x7f403c0ab0e0 con 0x7f403c13c480 2024-03-25T18:35:02.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.205+0000 7f40409ea640 1 -- 172.21.15.3:0/4048786672 >> v1:172.21.15.3:6790/0 conn(0x7f403c0b4ff0 legacy=0x7f403c13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:02.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.205+0000 7f40409ea640 1 -- 172.21.15.3:0/4048786672 >> v1:172.21.15.3:6789/0 conn(0x7f403c0b4670 legacy=0x7f403c1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:02.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.205+0000 7f40409ea640 10 monclient: _finish_hunting 0 2024-03-25T18:35:02.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.205+0000 7f40409ea640 1 monclient: found mon.noname-c 2024-03-25T18:35:02.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.205+0000 7f40409ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:02.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.205+0000 7f40409ea640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:35:02.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.205+0000 7f40409ea640 1 -- 172.21.15.3:0/4048786672 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f403c0a9b70 con 0x7f403c13c480 2024-03-25T18:35:02.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.205+0000 7f40409ea640 10 monclient: _finish_auth 0 2024-03-25T18:35:02.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.205+0000 7f40409ea640 10 monclient: _check_auth_tickets 2024-03-25T18:35:02.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.205+0000 7f40409ea640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:02.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.205+0000 7f40409ea640 1 -- 172.21.15.3:0/4048786672 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4034001930 con 0x7f403c13c480 2024-03-25T18:35:02.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.205+0000 7f40409ea640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:02.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.205+0000 7f40409ea640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:02.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.205+0000 7f40409ea640 10 monclient: dump: 2024-03-25T18:35:02.208 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:02.208 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:02.208 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:02.208 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:02.208 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:02.208 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:02.209 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:02.209 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:02.209 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:02.209 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:02.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.205+0000 7f4043c50640 5 monclient: authenticate success, global_id 4627 2024-03-25T18:35:02.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.205+0000 7f4043c50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:02.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.222+0000 7f40409ea640 1 -- 172.21.15.3:0/4048786672 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4034001a90 con 0x7f403c13c480 2024-03-25T18:35:02.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.222+0000 7f40409ea640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:02.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.222+0000 7f40409ea640 1 -- 172.21.15.3:0/4048786672 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4034001ee0 con 0x7f403c13c480 2024-03-25T18:35:02.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.222+0000 7f40409ea640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:02.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.222+0000 7f40409ea640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:02.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.222+0000 7f40409ea640 10 monclient: dump: 2024-03-25T18:35:02.224 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:02.224 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:02.224 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:02.224 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:02.224 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:02.224 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:02.224 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:02.224 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:02.225 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:02.225 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:02.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.222+0000 7f4043c50640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:02.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.222+0000 7f4043c50640 4 set_mon_vals no callback set 2024-03-25T18:35:02.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.223+0000 7f4043c50640 10 monclient: shutdown 2024-03-25T18:35:02.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.223+0000 7f4043c50640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:02.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.223+0000 7f4043c50640 1 -- 172.21.15.3:0/4048786672 >> v1:172.21.15.16:6789/0 conn(0x7f403c13c480 legacy=0x7f403c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:02.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.223+0000 7f4043c50640 1 -- 172.21.15.3:0/4048786672 shutdown_connections 2024-03-25T18:35:02.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.223+0000 7f4043c50640 1 -- 172.21.15.3:0/4048786672 >> 172.21.15.3:0/4048786672 conn(0x7f403c0617a0 msgr2=0x7f403c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:02.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.223+0000 7f4043c50640 1 -- 172.21.15.3:0/4048786672 shutdown_connections 2024-03-25T18:35:02.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.223+0000 7f4043c50640 1 -- 172.21.15.3:0/4048786672 wait complete. 2024-03-25T18:35:02.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.224+0000 7f4043c50640 10 monclient: build_initial_monmap 2024-03-25T18:35:02.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.224+0000 7f4043c50640 10 monclient: monmap: 2024-03-25T18:35:02.228 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:35:02.228 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:02.228 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:35:02.228 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:35:02.228 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:02.228 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:35:02.228 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:35:02.228 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:35:02.229 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:35:02.229 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:02.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.224+0000 7f4043c50640 1 librados: starting msgr at 2024-03-25T18:35:02.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.224+0000 7f4043c50640 1 librados: starting objecter 2024-03-25T18:35:02.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.225+0000 7f4043c50640 1 Processor -- start 2024-03-25T18:35:02.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.225+0000 7f4043c50640 1 -- start start 2024-03-25T18:35:02.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.225+0000 7f4043c50640 1 librados: setting wanted keys 2024-03-25T18:35:02.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.225+0000 7f4043c50640 1 librados: calling monclient init 2024-03-25T18:35:02.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.225+0000 7f4043c50640 10 monclient: init 2024-03-25T18:35:02.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.225+0000 7f4043c50640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:02.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.225+0000 7f4043c50640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:35:02.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.225+0000 7f4043c50640 10 monclient(hunting): picked mon.noname-b con 0x7f403c0b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:35:02.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.225+0000 7f4043c50640 10 monclient(hunting): picked mon.noname-a con 0x7f403c0b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:35:02.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.225+0000 7f4043c50640 10 monclient(hunting): picked mon.noname-c con 0x7f403c13c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:35:02.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.225+0000 7f4043c50640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f403c1459b0 con 0x7f403c0b4ff0 2024-03-25T18:35:02.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.225+0000 7f4043c50640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f403c13d690 con 0x7f403c0b4670 2024-03-25T18:35:02.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.225+0000 7f4043c50640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f403c13d970 con 0x7f403c13c480 2024-03-25T18:35:02.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.225+0000 7f4043c50640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:02.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.225+0000 7f4043c50640 10 monclient(hunting): authenticate will time out at 7956.486071s 2024-03-25T18:35:02.235 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.234+0000 7f40421ed640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f403c13c480 0x7f403c1e2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:53418/0 (socket says 172.21.15.3:53418) 2024-03-25T18:35:02.235 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.234+0000 7f40421ed640 1 -- 172.21.15.3:0/3034173799 learned_addr learned my addr 172.21.15.3:0/3034173799 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:35:02.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.237+0000 7f40327fc640 10 client.?.objecter ms_handle_connect 0x7f403c0b4670 2024-03-25T18:35:02.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.237+0000 7f40327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:02.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.241+0000 7f40327fc640 10 client.?.objecter ms_handle_connect 0x7f403c13c480 2024-03-25T18:35:02.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.241+0000 7f40327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:02.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.250+0000 7f40327fc640 10 client.?.objecter ms_handle_connect 0x7f403c0b4ff0 2024-03-25T18:35:02.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.250+0000 7f40327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:02.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.251+0000 7f40327fc640 1 -- 172.21.15.3:0/3034173799 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 157669798 0 0) 0x7f403c13d970 con 0x7f403c13c480 2024-03-25T18:35:02.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.251+0000 7f40327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:02.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.251+0000 7f40327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:02.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.251+0000 7f40327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:02.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.251+0000 7f40327fc640 10 monclient(hunting): my global_id is 4634 2024-03-25T18:35:02.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.251+0000 7f40327fc640 1 -- 172.21.15.3:0/3034173799 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4020004eb0 con 0x7f403c13c480 2024-03-25T18:35:02.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.266+0000 7f40327fc640 1 -- 172.21.15.3:0/3034173799 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2980315671 0 0) 0x7f4020004eb0 con 0x7f403c13c480 2024-03-25T18:35:02.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.266+0000 7f40327fc640 1 -- 172.21.15.3:0/3034173799 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f403c13d970 con 0x7f403c13c480 2024-03-25T18:35:02.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.266+0000 7f40327fc640 1 -- 172.21.15.3:0/3034173799 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4034001ea0 con 0x7f403c13c480 2024-03-25T18:35:02.268 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.266+0000 7f40327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:02.268 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.266+0000 7f40327fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:02.268 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.266+0000 7f40327fc640 10 monclient(hunting): dump: 2024-03-25T18:35:02.268 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:02.268 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:02.268 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:02.268 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:02.268 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:02.268 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:02.268 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:02.268 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:02.269 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:02.269 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:02.269 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.266+0000 7f40327fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:02.269 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.267+0000 7f40327fc640 1 -- 172.21.15.3:0/3034173799 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4150917900 0 0) 0x7f403c13d690 con 0x7f403c0b4670 2024-03-25T18:35:02.269 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.267+0000 7f40327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:02.269 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.267+0000 7f40327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:02.269 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.267+0000 7f40327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:02.270 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.267+0000 7f40327fc640 10 monclient(hunting): my global_id is 4630 2024-03-25T18:35:02.270 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.267+0000 7f40327fc640 1 -- 172.21.15.3:0/3034173799 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4020004eb0 con 0x7f403c0b4670 2024-03-25T18:35:02.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.271+0000 7f40327fc640 1 -- 172.21.15.3:0/3034173799 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3521870925 0 0) 0x7f403c1459b0 con 0x7f403c0b4ff0 2024-03-25T18:35:02.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.271+0000 7f40327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:02.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.271+0000 7f40327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:02.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.271+0000 7f40327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:02.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.271+0000 7f40327fc640 10 monclient(hunting): my global_id is 4683 2024-03-25T18:35:02.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.271+0000 7f40327fc640 1 -- 172.21.15.3:0/3034173799 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f403c13d690 con 0x7f403c0b4ff0 2024-03-25T18:35:02.282 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.280+0000 7f40327fc640 1 -- 172.21.15.3:0/3034173799 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2688051540 0 0) 0x7f403c13d690 con 0x7f403c0b4ff0 2024-03-25T18:35:02.282 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.280+0000 7f40327fc640 1 -- 172.21.15.3:0/3034173799 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f403c1459b0 con 0x7f403c0b4ff0 2024-03-25T18:35:02.282 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.280+0000 7f40327fc640 1 -- 172.21.15.3:0/3034173799 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f402c001930 con 0x7f403c0b4ff0 2024-03-25T18:35:02.282 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.280+0000 7f40327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:02.282 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.280+0000 7f40327fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:02.282 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.280+0000 7f40327fc640 10 monclient(hunting): dump: 2024-03-25T18:35:02.282 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:02.282 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:02.282 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:02.283 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:02.283 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:02.283 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:02.283 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:02.283 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:02.283 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:02.283 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:02.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.282+0000 7f40327fc640 1 -- 172.21.15.3:0/3034173799 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 473606740 0 0) 0x7f403c13d970 con 0x7f403c13c480 2024-03-25T18:35:02.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.282+0000 7f40327fc640 1 -- 172.21.15.3:0/3034173799 >> v1:172.21.15.16:6789/0 conn(0x7f403c0b4670 legacy=0x7f403c1327e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:02.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.282+0000 7f40327fc640 1 -- 172.21.15.3:0/3034173799 >> v1:172.21.15.3:6789/0 conn(0x7f403c0b4ff0 legacy=0x7f403c13e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:02.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.283+0000 7f40327fc640 10 monclient: _finish_hunting 0 2024-03-25T18:35:02.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.283+0000 7f40327fc640 1 monclient: found mon.c 2024-03-25T18:35:02.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.283+0000 7f40327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:02.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.283+0000 7f40327fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:35:02.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.283+0000 7f40327fc640 1 -- 172.21.15.3:0/3034173799 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f403c13c910 con 0x7f403c13c480 2024-03-25T18:35:02.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.283+0000 7f40327fc640 10 monclient: _finish_auth 0 2024-03-25T18:35:02.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.283+0000 7f40327fc640 10 monclient: _check_auth_tickets 2024-03-25T18:35:02.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.283+0000 7f40327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:02.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.283+0000 7f4043c50640 5 monclient: authenticate success, global_id 4634 2024-03-25T18:35:02.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.283+0000 7f4043c50640 10 monclient: _renew_subs 2024-03-25T18:35:02.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.283+0000 7f4043c50640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:35:02.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.283+0000 7f4043c50640 1 -- 172.21.15.3:0/3034173799 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f403c145e40 con 0x7f403c13c480 2024-03-25T18:35:02.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.283+0000 7f4043c50640 10 client.4634.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:02.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.283+0000 7f4043c50640 10 monclient: _renew_subs 2024-03-25T18:35:02.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.283+0000 7f4043c50640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:35:02.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.283+0000 7f4043c50640 1 -- 172.21.15.3:0/3034173799 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f403c145620 con 0x7f403c13c480 2024-03-25T18:35:02.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.283+0000 7f4043c50640 1 librados: init done 2024-03-25T18:35:02.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.284+0000 7f4043c50640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:02.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.284+0000 7f4043c50640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:02.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.284+0000 7f4043c50640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:35:02.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.284+0000 7f4043c50640 1 -- 172.21.15.3:0/3034173799 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f403c06dbd0 con 0x7f403c13c480 2024-03-25T18:35:02.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.605+0000 7f40327fc640 1 -- 172.21.15.3:0/3034173799 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4034003c20 con 0x7f403c13c480 2024-03-25T18:35:02.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.605+0000 7f40327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:02.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.605+0000 7f40327fc640 1 -- 172.21.15.3:0/3034173799 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4034001590 con 0x7f403c13c480 2024-03-25T18:35:02.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.605+0000 7f40327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:02.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.605+0000 7f40327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:02.607 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.605+0000 7f40327fc640 10 monclient: dump: 2024-03-25T18:35:02.607 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:02.607 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:02.607 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:02.607 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:02.607 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:02.607 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:02.607 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:02.607 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:02.607 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:02.607 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:02.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.605+0000 7f4033fff640 4 set_mon_vals no callback set 2024-03-25T18:35:02.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.605+0000 7f40327fc640 1 -- 172.21.15.3:0/3034173799 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f40340040b0 con 0x7f403c13c480 2024-03-25T18:35:02.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.606+0000 7f40419ec640 10 monclient: get_auth_request con 0x7f402006e020 auth_method 0 2024-03-25T18:35:02.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.606+0000 7f40327fc640 10 client.4634.objecter ms_handle_connect 0x7f402006e020 2024-03-25T18:35:02.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.613+0000 7f40327fc640 1 -- 172.21.15.3:0/3034173799 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 147098350 0 0) 0x7f4034054b70 con 0x7f403c13c480 2024-03-25T18:35:02.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.613+0000 7f40327fc640 10 client.4634.objecter ms_dispatch 0x7f403c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:35:02.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.613+0000 7f40327fc640 3 client.4634.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:35:02.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.613+0000 7f40327fc640 3 client.4634.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:35:02.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.613+0000 7f40327fc640 20 client.4634.objecter dump_active .. 0 homeless 2024-03-25T18:35:02.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.613+0000 7f40327fc640 1 -- 172.21.15.3:0/3034173799 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4034050680 con 0x7f403c13c480 2024-03-25T18:35:02.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.613+0000 7f40327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:02.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.613+0000 7f40327fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:02.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.865+0000 7f4043c50640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:02.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.865+0000 7f4043c50640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:02.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.865+0000 7f4043c50640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:35:02.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.865+0000 7f4043c50640 1 -- 172.21.15.3:0/3034173799 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f403c145b60 con 0x7f403c13c480 2024-03-25T18:35:02.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.914+0000 7f40327fc640 1 -- 172.21.15.3:0/3034173799 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1453446653) 0x7f403c145b60 con 0x7f403c13c480 2024-03-25T18:35:02.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.914+0000 7f40327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:02.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.914+0000 7f40327fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:02.916 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:35:02.916 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:25:09.249215+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:09.249215+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:02.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.916+0000 7f4043c50640 10 librados: watch_flush enter 2024-03-25T18:35:02.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.916+0000 7f4043c50640 10 librados: watch_flush exit 2024-03-25T18:35:02.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.916+0000 7f4043c50640 20 client.4634.objecter shutdown clearing up homeless session... 2024-03-25T18:35:02.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.916+0000 7f4043c50640 10 client.4634.objecter successfully canceled tick 2024-03-25T18:35:02.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.916+0000 7f4043c50640 1 -- 172.21.15.3:0/3034173799 >> v1:172.21.15.3:6800/4195140966 conn(0x7f402006e020 legacy=0x7f40200704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:02.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.916+0000 7f4043c50640 10 monclient: shutdown 2024-03-25T18:35:02.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.916+0000 7f4043c50640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:02.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.916+0000 7f4043c50640 1 -- 172.21.15.3:0/3034173799 >> v1:172.21.15.3:6790/0 conn(0x7f403c13c480 legacy=0x7f403c1e2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:02.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.916+0000 7f4043c50640 1 -- 172.21.15.3:0/3034173799 shutdown_connections 2024-03-25T18:35:02.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.916+0000 7f4043c50640 1 -- 172.21.15.3:0/3034173799 >> 172.21.15.3:0/3034173799 conn(0x7f403c0617a0 msgr2=0x7f403c140670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:02.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.916+0000 7f4043c50640 1 -- 172.21.15.3:0/3034173799 shutdown_connections 2024-03-25T18:35:02.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.916+0000 7f4043c50640 1 -- 172.21.15.3:0/3034173799 wait complete. 2024-03-25T18:35:02.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:02.916+0000 7f4043c50640 1 librados: shutdown 2024-03-25T18:35:02.931 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:03.326 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:10:17.055 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:03.381 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:10:17.110 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:03.413 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:10:17.142 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:03.428 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:10:17.120 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:03.435 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:10:17.164 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:03.436 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:10:17.128 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:03.450 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:10:17.180 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:03.472 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:10:17.164 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:03.484 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:10:17.214 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:09.739 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:09.854 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.853+0000 7f46dab97640 10 monclient: get_monmap_and_config 2024-03-25T18:35:09.854 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.853+0000 7f46dab97640 10 monclient: build_initial_monmap 2024-03-25T18:35:09.854 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.853+0000 7f46dab97640 10 monclient: monmap: 2024-03-25T18:35:09.854 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:35:09.855 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:09.855 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:35:09.854421+0000 2024-03-25T18:35:09.855 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:35:09.854421+0000 2024-03-25T18:35:09.855 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:09.855 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:09.855 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:35:09.855 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:35:09.855 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:35:09.855 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:09.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.854+0000 7f46dab97640 1 Processor -- start 2024-03-25T18:35:09.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.854+0000 7f46dab97640 1 -- start start 2024-03-25T18:35:09.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.854+0000 7f46dab97640 10 monclient: init 2024-03-25T18:35:09.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.855+0000 7f46dab97640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:09.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.855+0000 7f46dab97640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:35:09.858 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.855+0000 7f46dab97640 10 monclient(hunting): picked mon.noname-b con 0x7f46d40b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:35:09.858 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.855+0000 7f46dab97640 10 monclient(hunting): picked mon.noname-a con 0x7f46d40b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:35:09.858 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.855+0000 7f46dab97640 10 monclient(hunting): picked mon.noname-c con 0x7f46d413c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:35:09.858 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.855+0000 7f46dab97640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d40ab0e0 con 0x7f46d40b4670 2024-03-25T18:35:09.858 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.855+0000 7f46dab97640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d40ab3c0 con 0x7f46d413c480 2024-03-25T18:35:09.858 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.855+0000 7f46dab97640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d40a9f30 con 0x7f46d40b4ff0 2024-03-25T18:35:09.858 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.855+0000 7f46dab97640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:09.858 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.855+0000 7f46dab97640 10 monclient(hunting): authenticate will time out at 7964.115849s 2024-03-25T18:35:09.864 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.862+0000 7f46d8933640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f46d40b4ff0 0x7f46d413bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:42810/0 (socket says 172.21.15.3:42810) 2024-03-25T18:35:09.864 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.862+0000 7f46d8933640 1 -- 172.21.15.3:0/3854195106 learned_addr learned my addr 172.21.15.3:0/3854195106 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:35:09.944 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.943+0000 7f46d37fe640 1 -- 172.21.15.3:0/3854195106 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2346267374 0 0) 0x7f46d40ab0e0 con 0x7f46d40b4670 2024-03-25T18:35:09.944 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.943+0000 7f46d37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:35:09.944 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.943+0000 7f46d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:09.945 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.943+0000 7f46d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:09.945 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.943+0000 7f46d37fe640 10 monclient(hunting): my global_id is 4686 2024-03-25T18:35:09.945 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.943+0000 7f46d37fe640 1 -- 172.21.15.3:0/3854195106 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46bc003520 con 0x7f46d40b4670 2024-03-25T18:35:09.962 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.961+0000 7f46d37fe640 1 -- 172.21.15.3:0/3854195106 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2663589620 0 0) 0x7f46bc003520 con 0x7f46d40b4670 2024-03-25T18:35:09.962 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.961+0000 7f46d37fe640 1 -- 172.21.15.3:0/3854195106 >> v1:172.21.15.3:6790/0 conn(0x7f46d40b4ff0 legacy=0x7f46d413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:09.962 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.961+0000 7f46d37fe640 1 -- 172.21.15.3:0/3854195106 >> v1:172.21.15.16:6789/0 conn(0x7f46d413c480 legacy=0x7f46d414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:09.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.961+0000 7f46d37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:35:09.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.961+0000 7f46d37fe640 1 monclient: found mon.noname-a 2024-03-25T18:35:09.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.961+0000 7f46d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:09.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.961+0000 7f46d37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:35:09.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.961+0000 7f46d37fe640 1 -- 172.21.15.3:0/3854195106 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d40a9b70 con 0x7f46d40b4670 2024-03-25T18:35:09.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.961+0000 7f46d37fe640 10 monclient: _finish_auth 0 2024-03-25T18:35:09.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.961+0000 7f46d37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:35:09.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.961+0000 7f46d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:09.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.961+0000 7f46d37fe640 1 -- 172.21.15.3:0/3854195106 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f46c4001690 con 0x7f46d40b4670 2024-03-25T18:35:09.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.961+0000 7f46d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:09.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.961+0000 7f46d37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:09.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.961+0000 7f46d37fe640 10 monclient: dump: 2024-03-25T18:35:09.964 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:09.964 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:09.964 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:09.964 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:09.964 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:09.964 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:09.964 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:09.965 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:09.965 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:09.965 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:09.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.961+0000 7f46dab97640 5 monclient: authenticate success, global_id 4686 2024-03-25T18:35:09.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.961+0000 7f46dab97640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:09.981 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.980+0000 7f46d37fe640 1 -- 172.21.15.3:0/3854195106 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46c40017f0 con 0x7f46d40b4670 2024-03-25T18:35:09.981 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.980+0000 7f46d37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:09.981 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.980+0000 7f46d37fe640 1 -- 172.21.15.3:0/3854195106 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f46c4001c00 con 0x7f46d40b4670 2024-03-25T18:35:09.981 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.980+0000 7f46d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:09.981 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.980+0000 7f46d37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:09.982 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.980+0000 7f46d37fe640 10 monclient: dump: 2024-03-25T18:35:09.982 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:09.982 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:09.982 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:09.982 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:09.982 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:09.982 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:09.982 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:09.983 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:09.983 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:09.983 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:09.983 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.980+0000 7f46dab97640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:09.983 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.980+0000 7f46dab97640 4 set_mon_vals no callback set 2024-03-25T18:35:09.986 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.980+0000 7f46dab97640 10 monclient: shutdown 2024-03-25T18:35:09.986 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.980+0000 7f46dab97640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:09.986 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.980+0000 7f46dab97640 1 -- 172.21.15.3:0/3854195106 >> v1:172.21.15.3:6789/0 conn(0x7f46d40b4670 legacy=0x7f46d41425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:09.986 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.980+0000 7f46dab97640 1 -- 172.21.15.3:0/3854195106 shutdown_connections 2024-03-25T18:35:09.986 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.980+0000 7f46dab97640 1 -- 172.21.15.3:0/3854195106 >> 172.21.15.3:0/3854195106 conn(0x7f46d40617a0 msgr2=0x7f46d4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:09.986 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.981+0000 7f46dab97640 1 -- 172.21.15.3:0/3854195106 shutdown_connections 2024-03-25T18:35:09.986 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.981+0000 7f46dab97640 1 -- 172.21.15.3:0/3854195106 wait complete. 2024-03-25T18:35:09.986 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.981+0000 7f46dab97640 10 monclient: build_initial_monmap 2024-03-25T18:35:09.987 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.981+0000 7f46dab97640 10 monclient: monmap: 2024-03-25T18:35:09.987 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:35:09.987 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:09.987 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:35:09.987 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:35:09.987 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:09.987 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:35:09.987 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:35:09.987 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:35:09.988 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:35:09.988 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:09.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.981+0000 7f46dab97640 1 librados: starting msgr at 2024-03-25T18:35:09.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.981+0000 7f46dab97640 1 librados: starting objecter 2024-03-25T18:35:09.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.982+0000 7f46dab97640 1 Processor -- start 2024-03-25T18:35:09.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.982+0000 7f46dab97640 1 -- start start 2024-03-25T18:35:09.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.982+0000 7f46dab97640 1 librados: setting wanted keys 2024-03-25T18:35:09.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.982+0000 7f46dab97640 1 librados: calling monclient init 2024-03-25T18:35:09.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.982+0000 7f46dab97640 10 monclient: init 2024-03-25T18:35:09.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.982+0000 7f46dab97640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:09.989 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.982+0000 7f46dab97640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:35:09.989 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.982+0000 7f46dab97640 10 monclient(hunting): picked mon.noname-c con 0x7f46d40b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:35:09.989 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.982+0000 7f46dab97640 10 monclient(hunting): picked mon.noname-a con 0x7f46d40b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:35:09.989 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.982+0000 7f46dab97640 10 monclient(hunting): picked mon.noname-b con 0x7f46d413c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:35:09.989 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.982+0000 7f46dab97640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d41dbf50 con 0x7f46d40b4ff0 2024-03-25T18:35:09.989 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.982+0000 7f46dab97640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d4144650 con 0x7f46d413c480 2024-03-25T18:35:09.989 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.982+0000 7f46dab97640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d4144930 con 0x7f46d40b4670 2024-03-25T18:35:09.989 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.982+0000 7f46dab97640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:09.989 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:09.982+0000 7f46dab97640 10 monclient(hunting): authenticate will time out at 7964.243120s 2024-03-25T18:35:10.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.001+0000 7f46d3fff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f46d40b4ff0 0x7f46d41db130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:45966/0 (socket says 172.21.15.3:45966) 2024-03-25T18:35:10.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.001+0000 7f46d3fff640 1 -- 172.21.15.3:0/3971188943 learned_addr learned my addr 172.21.15.3:0/3971188943 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:35:10.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.002+0000 7f46d17fa640 10 client.?.objecter ms_handle_connect 0x7f46d40b4ff0 2024-03-25T18:35:10.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.002+0000 7f46d17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:10.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.004+0000 7f46d3fff640 1 -- 172.21.15.3:0/3971188943 >> v1:172.21.15.3:6789/0 conn(0x7f46d40b4ff0 legacy=0x7f46d41db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T18:35:10.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.004+0000 7f46d3fff640 1 -- 172.21.15.3:0/3971188943 >> v1:172.21.15.3:6789/0 conn(0x7f46d40b4ff0 legacy=0x7f46d41db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:35:10.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.004+0000 7f46d3fff640 1 --1- 172.21.15.3:0/3971188943 >> v1:172.21.15.3:6789/0 conn(0x7f46d40b4ff0 0x7f46d41db130 :-1 s=OPENED pgs=72599 cs=1 l=1).handle_message read tag failed 2024-03-25T18:35:10.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.004+0000 7f46d3fff640 1 --1- 172.21.15.3:0/3971188943 >> v1:172.21.15.3:6789/0 conn(0x7f46d40b4ff0 0x7f46d41db130 :-1 s=OPENED pgs=72599 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:35:10.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.004+0000 7f46d17fa640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.3:6789/0 2024-03-25T18:35:10.014 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.013+0000 7f46d17fa640 10 client.?.objecter ms_handle_connect 0x7f46d413c480 2024-03-25T18:35:10.015 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.013+0000 7f46d17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:10.028 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.027+0000 7f46d17fa640 10 client.?.objecter ms_handle_connect 0x7f46d40b4670 2024-03-25T18:35:10.029 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.027+0000 7f46d17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:10.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.110+0000 7f46d17fa640 1 -- 172.21.15.3:0/3971188943 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 335599618 0 0) 0x7f46d4144930 con 0x7f46d40b4670 2024-03-25T18:35:10.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.110+0000 7f46d17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:35:10.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.110+0000 7f46d17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:10.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.110+0000 7f46d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:10.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.110+0000 7f46d17fa640 10 monclient(hunting): my global_id is 4640 2024-03-25T18:35:10.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.110+0000 7f46d17fa640 1 -- 172.21.15.3:0/3971188943 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46c0004eb0 con 0x7f46d40b4670 2024-03-25T18:35:10.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.120+0000 7f46d17fa640 1 -- 172.21.15.3:0/3971188943 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3437132538 0 0) 0x7f46d4144650 con 0x7f46d413c480 2024-03-25T18:35:10.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.120+0000 7f46d17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:35:10.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.120+0000 7f46d17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:10.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.120+0000 7f46d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:10.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.120+0000 7f46d17fa640 10 monclient(hunting): my global_id is 4633 2024-03-25T18:35:10.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.120+0000 7f46d17fa640 1 -- 172.21.15.3:0/3971188943 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46d4144930 con 0x7f46d413c480 2024-03-25T18:35:10.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.151+0000 7f46d17fa640 1 -- 172.21.15.3:0/3971188943 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1143397807 0 0) 0x7f46c0004eb0 con 0x7f46d40b4670 2024-03-25T18:35:10.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.152+0000 7f46d17fa640 1 -- 172.21.15.3:0/3971188943 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46d4144650 con 0x7f46d40b4670 2024-03-25T18:35:10.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.152+0000 7f46d17fa640 1 -- 172.21.15.3:0/3971188943 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f46c80018a0 con 0x7f46d40b4670 2024-03-25T18:35:10.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.152+0000 7f46d17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:10.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.152+0000 7f46d17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:10.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.152+0000 7f46d17fa640 10 monclient(hunting): dump: 2024-03-25T18:35:10.154 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:10.154 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:10.154 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:10.154 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:10.154 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:10.154 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:10.154 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:10.155 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:10.155 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:10.155 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:10.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.152+0000 7f46d17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:10.156 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.155+0000 7f46d17fa640 1 -- 172.21.15.3:0/3971188943 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3026362357 0 0) 0x7f46d4144930 con 0x7f46d413c480 2024-03-25T18:35:10.156 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.155+0000 7f46d17fa640 1 -- 172.21.15.3:0/3971188943 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46c0004eb0 con 0x7f46d413c480 2024-03-25T18:35:10.156 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.155+0000 7f46d17fa640 1 -- 172.21.15.3:0/3971188943 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f46cc001a60 con 0x7f46d413c480 2024-03-25T18:35:10.156 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.155+0000 7f46d17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:10.156 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.155+0000 7f46d17fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:10.156 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.155+0000 7f46d17fa640 10 monclient(hunting): dump: 2024-03-25T18:35:10.156 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:10.156 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:10.157 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:10.157 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:10.157 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:10.157 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:10.157 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:10.157 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:10.157 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:10.157 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:10.183 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.181+0000 7f46d17fa640 1 -- 172.21.15.3:0/3971188943 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2165804017 0 0) 0x7f46d4144650 con 0x7f46d40b4670 2024-03-25T18:35:10.183 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.181+0000 7f46d17fa640 1 -- 172.21.15.3:0/3971188943 >> v1:172.21.15.16:6789/0 conn(0x7f46d413c480 legacy=0x7f46d41db840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:10.183 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.181+0000 7f46d17fa640 1 -- 172.21.15.3:0/3971188943 >> v1:172.21.15.3:6789/0 conn(0x7f46d40b4ff0 legacy=0x7f46d41db130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:35:10.183 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.181+0000 7f46d17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:35:10.183 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.181+0000 7f46d17fa640 1 monclient: found mon.c 2024-03-25T18:35:10.183 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.181+0000 7f46d17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:10.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.181+0000 7f46d17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:35:10.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.181+0000 7f46d17fa640 1 -- 172.21.15.3:0/3971188943 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d413ed60 con 0x7f46d40b4670 2024-03-25T18:35:10.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.181+0000 7f46d17fa640 10 monclient: _finish_auth 0 2024-03-25T18:35:10.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.181+0000 7f46d17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:35:10.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.181+0000 7f46d17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:10.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.181+0000 7f46dab97640 5 monclient: authenticate success, global_id 4640 2024-03-25T18:35:10.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.181+0000 7f46dab97640 10 monclient: _renew_subs 2024-03-25T18:35:10.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.181+0000 7f46dab97640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:35:10.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.181+0000 7f46dab97640 1 -- 172.21.15.3:0/3971188943 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d4145460 con 0x7f46d40b4670 2024-03-25T18:35:10.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.181+0000 7f46dab97640 10 client.4640.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:10.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.181+0000 7f46dab97640 10 monclient: _renew_subs 2024-03-25T18:35:10.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.181+0000 7f46dab97640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:35:10.185 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.181+0000 7f46dab97640 1 -- 172.21.15.3:0/3971188943 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d4143840 con 0x7f46d40b4670 2024-03-25T18:35:10.185 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.181+0000 7f46dab97640 1 librados: init done 2024-03-25T18:35:10.185 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.182+0000 7f46dab97640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:10.185 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.182+0000 7f46dab97640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:10.185 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.182+0000 7f46dab97640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:35:10.185 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.182+0000 7f46dab97640 1 -- 172.21.15.3:0/3971188943 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d406dbd0 con 0x7f46d40b4670 2024-03-25T18:35:10.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.534+0000 7f46d17fa640 1 -- 172.21.15.3:0/3971188943 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46c8001d10 con 0x7f46d40b4670 2024-03-25T18:35:10.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.534+0000 7f46d17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:10.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.534+0000 7f46d17fa640 1 -- 172.21.15.3:0/3971188943 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f46c8004980 con 0x7f46d40b4670 2024-03-25T18:35:10.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.534+0000 7f46d17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:10.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.534+0000 7f46d17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:10.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.534+0000 7f46d17fa640 10 monclient: dump: 2024-03-25T18:35:10.536 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:10.536 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:10.536 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:10.536 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:10.536 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:10.537 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:10.537 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:10.537 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:10.537 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:10.537 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:10.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.534+0000 7f46d2ffd640 4 set_mon_vals no callback set 2024-03-25T18:35:10.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.535+0000 7f46d17fa640 1 -- 172.21.15.3:0/3971188943 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f46c801a9d0 con 0x7f46d40b4670 2024-03-25T18:35:10.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.536+0000 7f46d3fff640 10 monclient: get_auth_request con 0x7f46c006de40 auth_method 0 2024-03-25T18:35:10.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.536+0000 7f46d17fa640 10 client.4640.objecter ms_handle_connect 0x7f46c006de40 2024-03-25T18:35:10.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.542+0000 7f46d17fa640 1 -- 172.21.15.3:0/3971188943 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2800263165 0 0) 0x7f46c8054730 con 0x7f46d40b4670 2024-03-25T18:35:10.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.542+0000 7f46d17fa640 10 client.4640.objecter ms_dispatch 0x7f46d4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:35:10.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.542+0000 7f46d17fa640 3 client.4640.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:35:10.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.542+0000 7f46d17fa640 3 client.4640.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:35:10.544 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.542+0000 7f46d17fa640 20 client.4640.objecter dump_active .. 0 homeless 2024-03-25T18:35:10.544 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.542+0000 7f46d17fa640 1 -- 172.21.15.3:0/3971188943 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f46c8050070 con 0x7f46d40b4670 2024-03-25T18:35:10.544 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.542+0000 7f46d17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:10.544 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.542+0000 7f46d17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:10.814 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.813+0000 7f46dab97640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:10.814 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.813+0000 7f46dab97640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:10.814 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.813+0000 7f46dab97640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:35:10.814 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.813+0000 7f46dab97640 1 -- 172.21.15.3:0/3971188943 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46d41455f0 con 0x7f46d40b4670 2024-03-25T18:35:10.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.860+0000 7f46d17fa640 1 -- 172.21.15.3:0/3971188943 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 716832884) 0x7f46c8052170 con 0x7f46d40b4670 2024-03-25T18:35:10.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.860+0000 7f46d17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:10.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.860+0000 7f46d17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:10.861 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:35:10.862 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":11,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:35:08.249841+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:35:08.249841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:10.864 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.863+0000 7f46dab97640 10 librados: watch_flush enter 2024-03-25T18:35:10.864 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.863+0000 7f46dab97640 10 librados: watch_flush exit 2024-03-25T18:35:10.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.863+0000 7f46dab97640 20 client.4640.objecter shutdown clearing up homeless session... 2024-03-25T18:35:10.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.863+0000 7f46dab97640 10 client.4640.objecter successfully canceled tick 2024-03-25T18:35:10.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.863+0000 7f46dab97640 1 -- 172.21.15.3:0/3971188943 >> v1:172.21.15.3:6800/4195140966 conn(0x7f46c006de40 legacy=0x7f46c0070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:10.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.863+0000 7f46dab97640 10 monclient: shutdown 2024-03-25T18:35:10.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.863+0000 7f46dab97640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:10.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.863+0000 7f46dab97640 1 -- 172.21.15.3:0/3971188943 >> v1:172.21.15.3:6790/0 conn(0x7f46d40b4670 legacy=0x7f46d4144d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:10.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.863+0000 7f46dab97640 1 -- 172.21.15.3:0/3971188943 shutdown_connections 2024-03-25T18:35:10.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.863+0000 7f46dab97640 1 -- 172.21.15.3:0/3971188943 >> 172.21.15.3:0/3971188943 conn(0x7f46d40617a0 msgr2=0x7f46d4140860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:10.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.863+0000 7f46dab97640 1 -- 172.21.15.3:0/3971188943 shutdown_connections 2024-03-25T18:35:10.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.863+0000 7f46dab97640 1 -- 172.21.15.3:0/3971188943 wait complete. 2024-03-25T18:35:10.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:10.863+0000 7f46dab97640 1 librados: shutdown 2024-03-25T18:35:10.876 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:17.682 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:17.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.785+0000 7f95f1a93640 10 monclient: get_monmap_and_config 2024-03-25T18:35:17.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.785+0000 7f95f1a93640 10 monclient: build_initial_monmap 2024-03-25T18:35:17.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.785+0000 7f95f1a93640 10 monclient: monmap: 2024-03-25T18:35:17.786 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:35:17.786 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:17.786 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:35:17.786282+0000 2024-03-25T18:35:17.786 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:35:17.786282+0000 2024-03-25T18:35:17.786 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:17.786 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:17.786 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:35:17.786 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:35:17.786 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:35:17.787 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:17.788 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.785+0000 7f95f1a93640 1 Processor -- start 2024-03-25T18:35:17.788 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.786+0000 7f95f1a93640 1 -- start start 2024-03-25T18:35:17.788 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.786+0000 7f95f1a93640 10 monclient: init 2024-03-25T18:35:17.788 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.786+0000 7f95f1a93640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:17.788 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.786+0000 7f95f1a93640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:35:17.788 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.786+0000 7f95f1a93640 10 monclient(hunting): picked mon.noname-a con 0x7f95ec0b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:35:17.788 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.786+0000 7f95f1a93640 10 monclient(hunting): picked mon.noname-b con 0x7f95ec0b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:35:17.788 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.786+0000 7f95f1a93640 10 monclient(hunting): picked mon.noname-c con 0x7f95ec13c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:35:17.788 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.786+0000 7f95f1a93640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec0ab0e0 con 0x7f95ec0b4ff0 2024-03-25T18:35:17.788 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.786+0000 7f95f1a93640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec0ab3c0 con 0x7f95ec13c480 2024-03-25T18:35:17.788 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.786+0000 7f95f1a93640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec0a9f30 con 0x7f95ec0b4670 2024-03-25T18:35:17.789 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.786+0000 7f95f1a93640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:17.789 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.786+0000 7f95f1a93640 10 monclient(hunting): authenticate will time out at 7972.047006s 2024-03-25T18:35:17.789 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.787+0000 7f95ebfff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f95ec13c480 0x7f95ec14dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:38498/0 (socket says 172.21.15.3:38498) 2024-03-25T18:35:17.789 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.787+0000 7f95ebfff640 1 -- 172.21.15.3:0/856723688 learned_addr learned my addr 172.21.15.3:0/856723688 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:35:17.824 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.823+0000 7f95eaffd640 1 -- 172.21.15.3:0/856723688 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4027462241 0 0) 0x7f95ec0ab0e0 con 0x7f95ec0b4ff0 2024-03-25T18:35:17.824 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.823+0000 7f95eaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:17.824 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.823+0000 7f95eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:17.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.823+0000 7f95eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:17.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.823+0000 7f95eaffd640 10 monclient(hunting): my global_id is 4692 2024-03-25T18:35:17.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.823+0000 7f95eaffd640 1 -- 172.21.15.3:0/856723688 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95d8003520 con 0x7f95ec0b4ff0 2024-03-25T18:35:17.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.847+0000 7f95eaffd640 1 -- 172.21.15.3:0/856723688 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3069964474 0 0) 0x7f95d8003520 con 0x7f95ec0b4ff0 2024-03-25T18:35:17.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.847+0000 7f95eaffd640 1 -- 172.21.15.3:0/856723688 >> v1:172.21.15.3:6790/0 conn(0x7f95ec0b4670 legacy=0x7f95ec1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:17.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.847+0000 7f95eaffd640 1 -- 172.21.15.3:0/856723688 >> v1:172.21.15.16:6789/0 conn(0x7f95ec13c480 legacy=0x7f95ec14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:17.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.847+0000 7f95eaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:35:17.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.847+0000 7f95eaffd640 1 monclient: found mon.noname-a 2024-03-25T18:35:17.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.847+0000 7f95eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:17.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.847+0000 7f95eaffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:35:17.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.847+0000 7f95eaffd640 1 -- 172.21.15.3:0/856723688 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ec0a9b70 con 0x7f95ec0b4ff0 2024-03-25T18:35:17.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.847+0000 7f95eaffd640 10 monclient: _finish_auth 0 2024-03-25T18:35:17.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.847+0000 7f95eaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:35:17.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.847+0000 7f95eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:17.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.847+0000 7f95eaffd640 1 -- 172.21.15.3:0/856723688 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f95dc001690 con 0x7f95ec0b4ff0 2024-03-25T18:35:17.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.847+0000 7f95eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:17.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.848+0000 7f95eaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:17.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.848+0000 7f95eaffd640 10 monclient: dump: 2024-03-25T18:35:17.850 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:17.850 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:17.850 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:17.850 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:17.850 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:17.850 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:17.850 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:17.850 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:17.850 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:17.850 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:17.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.848+0000 7f95f1a93640 5 monclient: authenticate success, global_id 4692 2024-03-25T18:35:17.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.848+0000 7f95f1a93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:17.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.868+0000 7f95eaffd640 1 -- 172.21.15.3:0/856723688 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95dc0017f0 con 0x7f95ec0b4ff0 2024-03-25T18:35:17.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.868+0000 7f95eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:17.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.868+0000 7f95eaffd640 1 -- 172.21.15.3:0/856723688 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f95dc001c40 con 0x7f95ec0b4ff0 2024-03-25T18:35:17.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.868+0000 7f95f1a93640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:17.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.868+0000 7f95f1a93640 4 set_mon_vals no callback set 2024-03-25T18:35:17.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.868+0000 7f95eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:17.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.868+0000 7f95eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:17.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.868+0000 7f95eaffd640 10 monclient: dump: 2024-03-25T18:35:17.870 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:17.870 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:17.870 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:17.870 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:17.870 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:17.870 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:17.870 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:17.870 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:17.871 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:17.871 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:17.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.869+0000 7f95f1a93640 10 monclient: shutdown 2024-03-25T18:35:17.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.869+0000 7f95f1a93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:17.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.869+0000 7f95f1a93640 1 -- 172.21.15.3:0/856723688 >> v1:172.21.15.3:6789/0 conn(0x7f95ec0b4ff0 legacy=0x7f95ec13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:17.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.869+0000 7f95f1a93640 1 -- 172.21.15.3:0/856723688 shutdown_connections 2024-03-25T18:35:17.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.869+0000 7f95f1a93640 1 -- 172.21.15.3:0/856723688 >> 172.21.15.3:0/856723688 conn(0x7f95ec0617a0 msgr2=0x7f95ec061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:17.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.869+0000 7f95f1a93640 1 -- 172.21.15.3:0/856723688 shutdown_connections 2024-03-25T18:35:17.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.869+0000 7f95f1a93640 1 -- 172.21.15.3:0/856723688 wait complete. 2024-03-25T18:35:17.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.870+0000 7f95f1a93640 10 monclient: build_initial_monmap 2024-03-25T18:35:17.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.870+0000 7f95f1a93640 10 monclient: monmap: 2024-03-25T18:35:17.872 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:35:17.872 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:17.872 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:35:17.873 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:35:17.873 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:17.873 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:35:17.873 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:35:17.873 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:35:17.873 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:35:17.873 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:17.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.871+0000 7f95f1a93640 1 librados: starting msgr at 2024-03-25T18:35:17.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.871+0000 7f95f1a93640 1 librados: starting objecter 2024-03-25T18:35:17.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.871+0000 7f95f1a93640 1 Processor -- start 2024-03-25T18:35:17.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.871+0000 7f95f1a93640 1 -- start start 2024-03-25T18:35:17.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.871+0000 7f95f1a93640 1 librados: setting wanted keys 2024-03-25T18:35:17.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.871+0000 7f95f1a93640 1 librados: calling monclient init 2024-03-25T18:35:17.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.871+0000 7f95f1a93640 10 monclient: init 2024-03-25T18:35:17.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.872+0000 7f95f1a93640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:17.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.872+0000 7f95f1a93640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:35:17.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.872+0000 7f95f1a93640 10 monclient(hunting): picked mon.noname-a con 0x7f95ec0b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:35:17.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.872+0000 7f95f1a93640 10 monclient(hunting): picked mon.noname-c con 0x7f95ec0b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:35:17.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.872+0000 7f95f1a93640 10 monclient(hunting): picked mon.noname-b con 0x7f95ec13c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:35:17.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.872+0000 7f95f1a93640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec0b0b90 con 0x7f95ec0b4670 2024-03-25T18:35:17.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.872+0000 7f95f1a93640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec0ae6d0 con 0x7f95ec13c480 2024-03-25T18:35:17.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.872+0000 7f95f1a93640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec14e1f0 con 0x7f95ec0b4ff0 2024-03-25T18:35:17.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.872+0000 7f95f1a93640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:17.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.872+0000 7f95f1a93640 10 monclient(hunting): authenticate will time out at 7972.132742s 2024-03-25T18:35:17.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.884+0000 7f95ebfff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f95ec13c480 0x7f95ec0af300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:38534/0 (socket says 172.21.15.3:38534) 2024-03-25T18:35:17.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.884+0000 7f95ebfff640 1 -- 172.21.15.3:0/3275622331 learned_addr learned my addr 172.21.15.3:0/3275622331 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:35:17.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.885+0000 7f95e8ff9640 10 client.?.objecter ms_handle_connect 0x7f95ec13c480 2024-03-25T18:35:17.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.885+0000 7f95e8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:17.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.892+0000 7f95e8ff9640 10 client.?.objecter ms_handle_connect 0x7f95ec0b4ff0 2024-03-25T18:35:17.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.892+0000 7f95e8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:17.896 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.895+0000 7f95e8ff9640 1 -- 172.21.15.3:0/3275622331 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3402851547 0 0) 0x7f95ec0ae6d0 con 0x7f95ec13c480 2024-03-25T18:35:17.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.895+0000 7f95e8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:35:17.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.895+0000 7f95e8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:17.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.895+0000 7f95e8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:17.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.895+0000 7f95e8ff9640 10 monclient(hunting): my global_id is 4639 2024-03-25T18:35:17.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.895+0000 7f95e8ff9640 1 -- 172.21.15.3:0/3275622331 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95cc004eb0 con 0x7f95ec13c480 2024-03-25T18:35:17.905 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.904+0000 7f95e8ff9640 10 client.?.objecter ms_handle_connect 0x7f95ec0b4670 2024-03-25T18:35:17.905 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.904+0000 7f95e8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:17.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.907+0000 7f95e8ff9640 1 -- 172.21.15.3:0/3275622331 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2381713151 0 0) 0x7f95ec14e1f0 con 0x7f95ec0b4ff0 2024-03-25T18:35:17.909 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.907+0000 7f95e8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:35:17.909 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.907+0000 7f95e8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:17.909 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.907+0000 7f95e8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:17.909 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.907+0000 7f95e8ff9640 10 monclient(hunting): my global_id is 4646 2024-03-25T18:35:17.909 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.907+0000 7f95e8ff9640 1 -- 172.21.15.3:0/3275622331 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95ec0ae6d0 con 0x7f95ec0b4ff0 2024-03-25T18:35:17.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.910+0000 7f95e8ff9640 1 -- 172.21.15.3:0/3275622331 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1257666869 0 0) 0x7f95cc004eb0 con 0x7f95ec13c480 2024-03-25T18:35:17.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.910+0000 7f95e8ff9640 1 -- 172.21.15.3:0/3275622331 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95ec14e1f0 con 0x7f95ec13c480 2024-03-25T18:35:17.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.910+0000 7f95e8ff9640 1 -- 172.21.15.3:0/3275622331 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f95e4001b10 con 0x7f95ec13c480 2024-03-25T18:35:17.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.910+0000 7f95e8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:17.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.910+0000 7f95e8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:35:17.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.910+0000 7f95e8ff9640 10 monclient(hunting): dump: 2024-03-25T18:35:17.911 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:17.911 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:17.911 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:17.911 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:17.912 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:17.912 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:17.912 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:17.912 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:17.912 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:17.912 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:17.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.910+0000 7f95e8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:17.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.917+0000 7f95e8ff9640 1 -- 172.21.15.3:0/3275622331 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1096088423 0 0) 0x7f95ec0b0b90 con 0x7f95ec0b4670 2024-03-25T18:35:17.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.917+0000 7f95e8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:35:17.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.917+0000 7f95e8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:17.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.917+0000 7f95e8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:17.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.917+0000 7f95e8ff9640 10 monclient(hunting): my global_id is 4695 2024-03-25T18:35:17.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.917+0000 7f95e8ff9640 1 -- 172.21.15.3:0/3275622331 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95cc004eb0 con 0x7f95ec0b4670 2024-03-25T18:35:17.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.918+0000 7f95e8ff9640 1 -- 172.21.15.3:0/3275622331 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2429994596 0 0) 0x7f95ec14e1f0 con 0x7f95ec13c480 2024-03-25T18:35:17.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.918+0000 7f95e8ff9640 1 -- 172.21.15.3:0/3275622331 >> v1:172.21.15.3:6790/0 conn(0x7f95ec0b4ff0 legacy=0x7f95ec0aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:17.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.918+0000 7f95e8ff9640 1 -- 172.21.15.3:0/3275622331 >> v1:172.21.15.3:6789/0 conn(0x7f95ec0b4670 legacy=0x7f95ec0b1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:17.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.918+0000 7f95e8ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:35:17.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.918+0000 7f95e8ff9640 1 monclient: found mon.b 2024-03-25T18:35:17.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.918+0000 7f95e8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:17.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.918+0000 7f95e8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:35:17.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.918+0000 7f95e8ff9640 1 -- 172.21.15.3:0/3275622331 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ec14e380 con 0x7f95ec13c480 2024-03-25T18:35:17.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.918+0000 7f95e8ff9640 10 monclient: _finish_auth 0 2024-03-25T18:35:17.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.918+0000 7f95e8ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:35:17.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.918+0000 7f95e8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:17.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.918+0000 7f95f1a93640 5 monclient: authenticate success, global_id 4639 2024-03-25T18:35:17.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.918+0000 7f95f1a93640 10 monclient: _renew_subs 2024-03-25T18:35:17.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.918+0000 7f95f1a93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:35:17.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.918+0000 7f95f1a93640 1 -- 172.21.15.3:0/3275622331 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95ec1e37f0 con 0x7f95ec13c480 2024-03-25T18:35:17.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.918+0000 7f95f1a93640 10 client.4639.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:17.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.918+0000 7f95f1a93640 10 monclient: _renew_subs 2024-03-25T18:35:17.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.918+0000 7f95f1a93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:35:17.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.918+0000 7f95f1a93640 1 -- 172.21.15.3:0/3275622331 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f95ec1e3b60 con 0x7f95ec13c480 2024-03-25T18:35:17.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.918+0000 7f95f1a93640 1 librados: init done 2024-03-25T18:35:17.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.919+0000 7f95f1a93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:17.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.919+0000 7f95f1a93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:17.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.919+0000 7f95f1a93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:35:17.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:17.919+0000 7f95f1a93640 1 -- 172.21.15.3:0/3275622331 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95ec0ae530 con 0x7f95ec13c480 2024-03-25T18:35:18.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.229+0000 7f95e8ff9640 1 -- 172.21.15.3:0/3275622331 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95e4002000 con 0x7f95ec13c480 2024-03-25T18:35:18.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.229+0000 7f95e8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:18.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.229+0000 7f95e8ff9640 1 -- 172.21.15.3:0/3275622331 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f95e4004b60 con 0x7f95ec13c480 2024-03-25T18:35:18.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.229+0000 7f95e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:18.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.229+0000 7f95e8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:18.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.229+0000 7f95e8ff9640 10 monclient: dump: 2024-03-25T18:35:18.230 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:18.230 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:18.231 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:18.231 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:18.231 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:18.231 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:18.231 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:18.231 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:18.231 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:18.231 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:18.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.229+0000 7f95ea7fc640 4 set_mon_vals no callback set 2024-03-25T18:35:18.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.230+0000 7f95e8ff9640 1 -- 172.21.15.3:0/3275622331 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f95e4004d40 con 0x7f95ec13c480 2024-03-25T18:35:18.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.230+0000 7f95eb7fe640 10 monclient: get_auth_request con 0x7f95cc06ddd0 auth_method 0 2024-03-25T18:35:18.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.231+0000 7f95e8ff9640 10 client.4639.objecter ms_handle_connect 0x7f95cc06ddd0 2024-03-25T18:35:18.236 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.235+0000 7f95e8ff9640 1 -- 172.21.15.3:0/3275622331 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2800263165 0 0) 0x7f95e4054900 con 0x7f95ec13c480 2024-03-25T18:35:18.236 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.235+0000 7f95e8ff9640 10 client.4639.objecter ms_dispatch 0x7f95ec000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:35:18.236 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.235+0000 7f95e8ff9640 3 client.4639.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:35:18.236 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.235+0000 7f95e8ff9640 3 client.4639.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:35:18.236 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.235+0000 7f95e8ff9640 20 client.4639.objecter dump_active .. 0 homeless 2024-03-25T18:35:18.236 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.235+0000 7f95e8ff9640 1 -- 172.21.15.3:0/3275622331 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f95e4050240 con 0x7f95ec13c480 2024-03-25T18:35:18.236 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.235+0000 7f95e8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:18.237 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.235+0000 7f95e8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:18.496 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.494+0000 7f95f1a93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:18.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.494+0000 7f95f1a93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:18.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.494+0000 7f95f1a93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:35:18.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.494+0000 7f95f1a93640 1 -- 172.21.15.3:0/3275622331 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95ec06dbd0 con 0x7f95ec13c480 2024-03-25T18:35:18.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.528+0000 7f95e8ff9640 1 -- 172.21.15.3:0/3275622331 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 716832884) 0x7f95e4052340 con 0x7f95ec13c480 2024-03-25T18:35:18.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.528+0000 7f95e8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:18.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.528+0000 7f95e8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:18.571 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:35:18.571 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":11,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:35:08.249841+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:35:08.249841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:18.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.530+0000 7f95f1a93640 10 librados: watch_flush enter 2024-03-25T18:35:18.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.530+0000 7f95f1a93640 10 librados: watch_flush exit 2024-03-25T18:35:18.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.530+0000 7f95f1a93640 20 client.4639.objecter shutdown clearing up homeless session... 2024-03-25T18:35:18.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.530+0000 7f95f1a93640 10 client.4639.objecter successfully canceled tick 2024-03-25T18:35:18.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.530+0000 7f95f1a93640 1 -- 172.21.15.3:0/3275622331 >> v1:172.21.15.3:6800/4195140966 conn(0x7f95cc06ddd0 legacy=0x7f95cc070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:18.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.530+0000 7f95f1a93640 10 monclient: shutdown 2024-03-25T18:35:18.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.530+0000 7f95f1a93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:18.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.530+0000 7f95f1a93640 1 -- 172.21.15.3:0/3275622331 >> v1:172.21.15.16:6789/0 conn(0x7f95ec13c480 legacy=0x7f95ec0af300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:18.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.530+0000 7f95f1a93640 1 -- 172.21.15.3:0/3275622331 shutdown_connections 2024-03-25T18:35:18.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.530+0000 7f95f1a93640 1 -- 172.21.15.3:0/3275622331 >> 172.21.15.3:0/3275622331 conn(0x7f95ec0617a0 msgr2=0x7f95ec1407a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:18.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.531+0000 7f95f1a93640 1 -- 172.21.15.3:0/3275622331 shutdown_connections 2024-03-25T18:35:18.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.531+0000 7f95f1a93640 1 -- 172.21.15.3:0/3275622331 wait complete. 2024-03-25T18:35:18.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:18.531+0000 7f95f1a93640 1 librados: shutdown 2024-03-25T18:35:18.574 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:23.424 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:10:37.153 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:23.429 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:10:37.158 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:23.460 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:10:37.189 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:23.480 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:10:37.173 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:23.487 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:10:37.179 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:23.488 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:10:37.217 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:23.494 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:10:37.223 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:23.535 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:10:37.227 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:23.537 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:10:37.266 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:25.380 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:25.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.497+0000 7f461fd3a640 10 monclient: get_monmap_and_config 2024-03-25T18:35:25.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.497+0000 7f461fd3a640 10 monclient: build_initial_monmap 2024-03-25T18:35:25.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.497+0000 7f461fd3a640 10 monclient: monmap: 2024-03-25T18:35:25.498 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:35:25.499 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:25.499 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:35:25.498334+0000 2024-03-25T18:35:25.499 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:35:25.498334+0000 2024-03-25T18:35:25.499 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:25.499 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:25.499 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:35:25.499 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:35:25.499 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:35:25.499 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:25.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.498+0000 7f461fd3a640 1 Processor -- start 2024-03-25T18:35:25.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.498+0000 7f461fd3a640 1 -- start start 2024-03-25T18:35:25.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.498+0000 7f461fd3a640 10 monclient: init 2024-03-25T18:35:25.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.499+0000 7f461fd3a640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:25.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.499+0000 7f461fd3a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:35:25.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.499+0000 7f461fd3a640 10 monclient(hunting): picked mon.noname-a con 0x7f46180b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:35:25.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.499+0000 7f461fd3a640 10 monclient(hunting): picked mon.noname-c con 0x7f46180b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:35:25.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.499+0000 7f461fd3a640 10 monclient(hunting): picked mon.noname-b con 0x7f461813c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:35:25.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.499+0000 7f461fd3a640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46180ab0e0 con 0x7f46180b4ff0 2024-03-25T18:35:25.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.499+0000 7f461fd3a640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46180ab3c0 con 0x7f46180b4670 2024-03-25T18:35:25.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.499+0000 7f461fd3a640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46180a9f30 con 0x7f461813c480 2024-03-25T18:35:25.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.499+0000 7f461fd3a640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:25.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.499+0000 7f461fd3a640 10 monclient(hunting): authenticate will time out at 7979.759921s 2024-03-25T18:35:25.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.514+0000 7f461d2d5640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f46180b4670 0x7f46181425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:34346/0 (socket says 172.21.15.3:34346) 2024-03-25T18:35:25.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.514+0000 7f461d2d5640 1 -- 172.21.15.3:0/181030676 learned_addr learned my addr 172.21.15.3:0/181030676 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:35:25.530 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.529+0000 7f461cad4640 1 -- 172.21.15.3:0/181030676 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3216746823 0 0) 0x7f46180ab0e0 con 0x7f46180b4ff0 2024-03-25T18:35:25.530 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.529+0000 7f461cad4640 10 monclient(hunting): _negotiate 2024-03-25T18:35:25.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.529+0000 7f461cad4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:25.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.529+0000 7f461cad4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:25.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.529+0000 7f461cad4640 10 monclient(hunting): my global_id is 4698 2024-03-25T18:35:25.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.529+0000 7f461cad4640 1 -- 172.21.15.3:0/181030676 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4604003520 con 0x7f46180b4ff0 2024-03-25T18:35:25.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.531+0000 7f461cad4640 1 -- 172.21.15.3:0/181030676 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2577860701 0 0) 0x7f46180ab3c0 con 0x7f46180b4670 2024-03-25T18:35:25.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.531+0000 7f461cad4640 10 monclient(hunting): _negotiate 2024-03-25T18:35:25.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.531+0000 7f461cad4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:25.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.531+0000 7f461cad4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:25.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.531+0000 7f461cad4640 10 monclient(hunting): my global_id is 4642 2024-03-25T18:35:25.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.531+0000 7f461cad4640 1 -- 172.21.15.3:0/181030676 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46180ab0e0 con 0x7f46180b4670 2024-03-25T18:35:25.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.534+0000 7f461cad4640 1 -- 172.21.15.3:0/181030676 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 353507128 0 0) 0x7f46180a9f30 con 0x7f461813c480 2024-03-25T18:35:25.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.534+0000 7f461cad4640 10 monclient(hunting): _negotiate 2024-03-25T18:35:25.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.534+0000 7f461cad4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:25.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.534+0000 7f461cad4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:25.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.534+0000 7f461cad4640 10 monclient(hunting): my global_id is 4649 2024-03-25T18:35:25.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.534+0000 7f461cad4640 1 -- 172.21.15.3:0/181030676 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46180ab3c0 con 0x7f461813c480 2024-03-25T18:35:25.542 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.540+0000 7f461cad4640 1 -- 172.21.15.3:0/181030676 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 736375653 0 0) 0x7f46180ab0e0 con 0x7f46180b4670 2024-03-25T18:35:25.542 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.540+0000 7f461cad4640 1 -- 172.21.15.3:0/181030676 >> v1:172.21.15.3:6790/0 conn(0x7f461813c480 legacy=0x7f461814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:25.542 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.540+0000 7f461cad4640 1 -- 172.21.15.3:0/181030676 >> v1:172.21.15.3:6789/0 conn(0x7f46180b4ff0 legacy=0x7f461813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:25.542 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.540+0000 7f461cad4640 10 monclient: _finish_hunting 0 2024-03-25T18:35:25.542 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.540+0000 7f461cad4640 1 monclient: found mon.noname-c 2024-03-25T18:35:25.542 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.540+0000 7f461cad4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:25.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.540+0000 7f461cad4640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:35:25.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.541+0000 7f461cad4640 1 -- 172.21.15.3:0/181030676 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46180a9b70 con 0x7f46180b4670 2024-03-25T18:35:25.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.541+0000 7f461cad4640 10 monclient: _finish_auth 0 2024-03-25T18:35:25.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.541+0000 7f461cad4640 10 monclient: _check_auth_tickets 2024-03-25T18:35:25.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.541+0000 7f461cad4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:25.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.541+0000 7f461cad4640 1 -- 172.21.15.3:0/181030676 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4608001670 con 0x7f46180b4670 2024-03-25T18:35:25.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.541+0000 7f461cad4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:25.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.541+0000 7f461cad4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:25.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.541+0000 7f461cad4640 10 monclient: dump: 2024-03-25T18:35:25.543 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:25.543 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:25.543 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:25.544 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:25.544 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:25.544 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:25.544 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:25.544 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:25.544 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:25.544 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:25.544 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.541+0000 7f461fd3a640 5 monclient: authenticate success, global_id 4642 2024-03-25T18:35:25.544 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.541+0000 7f461fd3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:25.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.569+0000 7f461cad4640 1 -- 172.21.15.3:0/181030676 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46080017d0 con 0x7f46180b4670 2024-03-25T18:35:25.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.570+0000 7f461cad4640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:25.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.570+0000 7f461cad4640 1 -- 172.21.15.3:0/181030676 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4608001be0 con 0x7f46180b4670 2024-03-25T18:35:25.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.570+0000 7f461fd3a640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:25.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.570+0000 7f461fd3a640 4 set_mon_vals no callback set 2024-03-25T18:35:25.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.570+0000 7f461cad4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:25.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.570+0000 7f461cad4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:25.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.570+0000 7f461cad4640 10 monclient: dump: 2024-03-25T18:35:25.571 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:25.571 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:25.571 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:25.572 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:25.572 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:25.572 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:25.572 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:25.572 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:25.572 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:25.572 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:25.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.570+0000 7f461fd3a640 10 monclient: shutdown 2024-03-25T18:35:25.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.570+0000 7f461fd3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:25.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.570+0000 7f461fd3a640 1 -- 172.21.15.3:0/181030676 >> v1:172.21.15.16:6789/0 conn(0x7f46180b4670 legacy=0x7f46181425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:25.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.570+0000 7f461fd3a640 1 -- 172.21.15.3:0/181030676 shutdown_connections 2024-03-25T18:35:25.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.570+0000 7f461fd3a640 1 -- 172.21.15.3:0/181030676 >> 172.21.15.3:0/181030676 conn(0x7f46180617a0 msgr2=0x7f4618061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:25.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.571+0000 7f461fd3a640 1 -- 172.21.15.3:0/181030676 shutdown_connections 2024-03-25T18:35:25.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.571+0000 7f461fd3a640 1 -- 172.21.15.3:0/181030676 wait complete. 2024-03-25T18:35:25.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.571+0000 7f461fd3a640 10 monclient: build_initial_monmap 2024-03-25T18:35:25.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.571+0000 7f461fd3a640 10 monclient: monmap: 2024-03-25T18:35:25.576 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:35:25.576 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:25.576 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:35:25.576 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:35:25.576 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:25.576 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:35:25.576 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:35:25.576 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:35:25.576 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:35:25.576 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:25.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.572+0000 7f461fd3a640 1 librados: starting msgr at 2024-03-25T18:35:25.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.572+0000 7f461fd3a640 1 librados: starting objecter 2024-03-25T18:35:25.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.572+0000 7f461fd3a640 1 Processor -- start 2024-03-25T18:35:25.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.572+0000 7f461fd3a640 1 -- start start 2024-03-25T18:35:25.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.572+0000 7f461fd3a640 1 librados: setting wanted keys 2024-03-25T18:35:25.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.572+0000 7f461fd3a640 1 librados: calling monclient init 2024-03-25T18:35:25.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.572+0000 7f461fd3a640 10 monclient: init 2024-03-25T18:35:25.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.573+0000 7f461fd3a640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:25.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.573+0000 7f461fd3a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:35:25.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.573+0000 7f461fd3a640 10 monclient(hunting): picked mon.noname-c con 0x7f46180b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:35:25.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.573+0000 7f461fd3a640 10 monclient(hunting): picked mon.noname-a con 0x7f46180b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:35:25.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.573+0000 7f461fd3a640 10 monclient(hunting): picked mon.noname-b con 0x7f461813c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:35:25.578 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.573+0000 7f461fd3a640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46181445b0 con 0x7f46180b4ff0 2024-03-25T18:35:25.578 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.573+0000 7f461fd3a640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46181461f0 con 0x7f461813c480 2024-03-25T18:35:25.578 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.573+0000 7f461fd3a640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4618144800 con 0x7f46180b4670 2024-03-25T18:35:25.578 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.573+0000 7f461fd3a640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:25.578 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.573+0000 7f461fd3a640 10 monclient(hunting): authenticate will time out at 7979.833708s 2024-03-25T18:35:25.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.583+0000 7f461dad6640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f46180b4670 0x7f46181db480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:40094/0 (socket says 172.21.15.3:40094) 2024-03-25T18:35:25.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.583+0000 7f461dad6640 1 -- 172.21.15.3:0/3768735073 learned_addr learned my addr 172.21.15.3:0/3768735073 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:35:25.585 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.584+0000 7f460e7fc640 10 client.?.objecter ms_handle_connect 0x7f46180b4670 2024-03-25T18:35:25.585 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.584+0000 7f460e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:25.588 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.587+0000 7f460e7fc640 10 client.?.objecter ms_handle_connect 0x7f46180b4ff0 2024-03-25T18:35:25.588 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.587+0000 7f460e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:25.591 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.590+0000 7f460e7fc640 10 client.?.objecter ms_handle_connect 0x7f461813c480 2024-03-25T18:35:25.591 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.590+0000 7f460e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:25.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.605+0000 7f460e7fc640 1 -- 172.21.15.3:0/3768735073 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2024340127 0 0) 0x7f46181445b0 con 0x7f46180b4ff0 2024-03-25T18:35:25.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.605+0000 7f460e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:25.607 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.605+0000 7f460e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:25.607 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.605+0000 7f460e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:25.607 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.605+0000 7f460e7fc640 10 monclient(hunting): my global_id is 4701 2024-03-25T18:35:25.607 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.605+0000 7f460e7fc640 1 -- 172.21.15.3:0/3768735073 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45fc004eb0 con 0x7f46180b4ff0 2024-03-25T18:35:25.607 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.606+0000 7f460e7fc640 1 -- 172.21.15.3:0/3768735073 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3322652199 0 0) 0x7f4618144800 con 0x7f46180b4670 2024-03-25T18:35:25.607 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.606+0000 7f460e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:25.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.606+0000 7f460e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:25.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.606+0000 7f460e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:25.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.606+0000 7f460e7fc640 10 monclient(hunting): my global_id is 4652 2024-03-25T18:35:25.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.606+0000 7f460e7fc640 1 -- 172.21.15.3:0/3768735073 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46181445b0 con 0x7f46180b4670 2024-03-25T18:35:25.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.613+0000 7f460e7fc640 1 -- 172.21.15.3:0/3768735073 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1685282417 0 0) 0x7f46181461f0 con 0x7f461813c480 2024-03-25T18:35:25.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.613+0000 7f460e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:25.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.613+0000 7f460e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:25.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.613+0000 7f460e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:25.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.613+0000 7f460e7fc640 10 monclient(hunting): my global_id is 4645 2024-03-25T18:35:25.615 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.613+0000 7f460e7fc640 1 -- 172.21.15.3:0/3768735073 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4618144800 con 0x7f461813c480 2024-03-25T18:35:25.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.621+0000 7f460e7fc640 1 -- 172.21.15.3:0/3768735073 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2588066507 0 0) 0x7f45fc004eb0 con 0x7f46180b4ff0 2024-03-25T18:35:25.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.621+0000 7f460e7fc640 1 -- 172.21.15.3:0/3768735073 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46181461f0 con 0x7f46180b4ff0 2024-03-25T18:35:25.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.621+0000 7f460e7fc640 1 -- 172.21.15.3:0/3768735073 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4608001ba0 con 0x7f46180b4ff0 2024-03-25T18:35:25.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.621+0000 7f460e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:25.623 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.621+0000 7f460e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:25.623 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.621+0000 7f460e7fc640 10 monclient(hunting): dump: 2024-03-25T18:35:25.623 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:25.623 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:25.623 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:25.623 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:25.623 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:25.623 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:25.623 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:25.623 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:25.623 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:25.623 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:25.623 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.621+0000 7f460e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:25.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.623+0000 7f460e7fc640 1 -- 172.21.15.3:0/3768735073 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1475496990 0 0) 0x7f46181445b0 con 0x7f46180b4670 2024-03-25T18:35:25.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.623+0000 7f460e7fc640 1 -- 172.21.15.3:0/3768735073 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45fc004eb0 con 0x7f46180b4670 2024-03-25T18:35:25.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.623+0000 7f460e7fc640 1 -- 172.21.15.3:0/3768735073 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4614001950 con 0x7f46180b4670 2024-03-25T18:35:25.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.623+0000 7f460e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:25.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.623+0000 7f460e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:25.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.623+0000 7f460e7fc640 10 monclient(hunting): dump: 2024-03-25T18:35:25.625 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:25.625 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:25.625 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:25.625 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:25.625 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:25.625 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:25.626 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:25.626 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:25.626 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:25.626 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:25.637 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.636+0000 7f460e7fc640 1 -- 172.21.15.3:0/3768735073 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4107309965 0 0) 0x7f4618144800 con 0x7f461813c480 2024-03-25T18:35:25.637 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.636+0000 7f460e7fc640 1 -- 172.21.15.3:0/3768735073 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46181445b0 con 0x7f461813c480 2024-03-25T18:35:25.638 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.636+0000 7f460e7fc640 1 -- 172.21.15.3:0/3768735073 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f4610001bd0 con 0x7f461813c480 2024-03-25T18:35:25.638 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.636+0000 7f460e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:25.638 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.636+0000 7f460e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:25.638 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.636+0000 7f460e7fc640 10 monclient(hunting): dump: 2024-03-25T18:35:25.638 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:25.638 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:25.638 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:25.638 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:25.638 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:25.639 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:25.639 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:25.639 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:25.639 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:25.639 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:25.641 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.638+0000 7f460e7fc640 1 -- 172.21.15.3:0/3768735073 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1017194127 0 0) 0x7f45fc004eb0 con 0x7f46180b4670 2024-03-25T18:35:25.641 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.638+0000 7f460e7fc640 1 -- 172.21.15.3:0/3768735073 >> v1:172.21.15.16:6789/0 conn(0x7f461813c480 legacy=0x7f46181e2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:25.641 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.638+0000 7f460e7fc640 1 -- 172.21.15.3:0/3768735073 >> v1:172.21.15.3:6789/0 conn(0x7f46180b4ff0 legacy=0x7f46181dbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:25.641 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.638+0000 7f460e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:35:25.641 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.638+0000 7f460e7fc640 1 monclient: found mon.c 2024-03-25T18:35:25.641 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.638+0000 7f460e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:25.641 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.638+0000 7f460e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:35:25.641 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.638+0000 7f460e7fc640 1 -- 172.21.15.3:0/3768735073 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4618144ae0 con 0x7f46180b4670 2024-03-25T18:35:25.641 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.638+0000 7f460e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:35:25.641 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.638+0000 7f460e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:35:25.642 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.638+0000 7f460e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:25.642 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.638+0000 7f461fd3a640 5 monclient: authenticate success, global_id 4652 2024-03-25T18:35:25.642 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.638+0000 7f461fd3a640 10 monclient: _renew_subs 2024-03-25T18:35:25.642 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.638+0000 7f461fd3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:35:25.642 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.638+0000 7f461fd3a640 1 -- 172.21.15.3:0/3768735073 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4618144dc0 con 0x7f46180b4670 2024-03-25T18:35:25.642 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.638+0000 7f461fd3a640 10 client.4652.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:25.642 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.638+0000 7f461fd3a640 10 monclient: _renew_subs 2024-03-25T18:35:25.642 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.638+0000 7f461fd3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:35:25.642 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.638+0000 7f461fd3a640 1 -- 172.21.15.3:0/3768735073 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46181466b0 con 0x7f46180b4670 2024-03-25T18:35:25.643 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.638+0000 7f461fd3a640 1 librados: init done 2024-03-25T18:35:25.643 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.639+0000 7f461fd3a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:25.643 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.639+0000 7f461fd3a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:25.643 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.639+0000 7f461fd3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:35:25.643 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.639+0000 7f461fd3a640 1 -- 172.21.15.3:0/3768735073 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46180688c0 con 0x7f46180b4670 2024-03-25T18:35:25.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.859+0000 7f460e7fc640 1 -- 172.21.15.3:0/3768735073 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4614003610 con 0x7f46180b4670 2024-03-25T18:35:25.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.859+0000 7f460e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:25.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.859+0000 7f460ffff640 4 set_mon_vals no callback set 2024-03-25T18:35:25.943 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.943+0000 7f460e7fc640 1 -- 172.21.15.3:0/3768735073 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f46140039e0 con 0x7f46180b4670 2024-03-25T18:35:25.943 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.943+0000 7f460e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:25.943 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.943+0000 7f460e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:25.943 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.943+0000 7f460e7fc640 10 monclient: dump: 2024-03-25T18:35:25.943 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:25.943 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:25.944 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:25.944 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:25.944 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:25.944 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:25.944 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:25.944 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:25.944 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:25.944 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:25.945 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.944+0000 7f460e7fc640 1 -- 172.21.15.3:0/3768735073 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f4614003bc0 con 0x7f46180b4670 2024-03-25T18:35:25.945 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.944+0000 7f461d2d5640 10 monclient: get_auth_request con 0x7f45fc06e110 auth_method 0 2024-03-25T18:35:25.945 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.945+0000 7f460e7fc640 10 client.4652.objecter ms_handle_connect 0x7f45fc06e110 2024-03-25T18:35:25.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.951+0000 7f460e7fc640 1 -- 172.21.15.3:0/3768735073 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2800263165 0 0) 0x7f46140547e0 con 0x7f46180b4670 2024-03-25T18:35:25.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.951+0000 7f460e7fc640 10 client.4652.objecter ms_dispatch 0x7f4618000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:35:25.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.951+0000 7f460e7fc640 3 client.4652.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:35:25.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.951+0000 7f460e7fc640 3 client.4652.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:35:25.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.951+0000 7f460e7fc640 20 client.4652.objecter dump_active .. 0 homeless 2024-03-25T18:35:25.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.951+0000 7f460e7fc640 1 -- 172.21.15.3:0/3768735073 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f46140502f0 con 0x7f46180b4670 2024-03-25T18:35:25.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.951+0000 7f460e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:25.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:25.951+0000 7f460e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:26.171 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:26.171+0000 7f461fd3a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:26.172 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:26.171+0000 7f461fd3a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:26.172 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:26.171+0000 7f461fd3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:35:26.172 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:26.171+0000 7f461fd3a640 1 -- 172.21.15.3:0/3768735073 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46181463d0 con 0x7f46180b4670 2024-03-25T18:35:26.210 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:26.210+0000 7f460e7fc640 1 -- 172.21.15.3:0/3768735073 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 716832884) 0x7f46181463d0 con 0x7f46180b4670 2024-03-25T18:35:26.210 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:26.210+0000 7f460e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:26.210 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:26.210+0000 7f460e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:26.210 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:35:26.210 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":11,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:35:08.249841+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:35:08.249841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:26.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:26.212+0000 7f461fd3a640 10 librados: watch_flush enter 2024-03-25T18:35:26.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:26.212+0000 7f461fd3a640 10 librados: watch_flush exit 2024-03-25T18:35:26.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:26.212+0000 7f461fd3a640 20 client.4652.objecter shutdown clearing up homeless session... 2024-03-25T18:35:26.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:26.212+0000 7f461fd3a640 10 client.4652.objecter successfully canceled tick 2024-03-25T18:35:26.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:26.212+0000 7f461fd3a640 1 -- 172.21.15.3:0/3768735073 >> v1:172.21.15.3:6800/4195140966 conn(0x7f45fc06e110 legacy=0x7f45fc0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:26.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:26.213+0000 7f461fd3a640 10 monclient: shutdown 2024-03-25T18:35:26.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:26.213+0000 7f461fd3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:26.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:26.213+0000 7f461fd3a640 1 -- 172.21.15.3:0/3768735073 >> v1:172.21.15.3:6790/0 conn(0x7f46180b4670 legacy=0x7f46181db480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:26.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:26.213+0000 7f461fd3a640 1 -- 172.21.15.3:0/3768735073 shutdown_connections 2024-03-25T18:35:26.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:26.213+0000 7f461fd3a640 1 -- 172.21.15.3:0/3768735073 >> 172.21.15.3:0/3768735073 conn(0x7f46180617a0 msgr2=0x7f46181405c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:26.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:26.213+0000 7f461fd3a640 1 -- 172.21.15.3:0/3768735073 shutdown_connections 2024-03-25T18:35:26.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:26.213+0000 7f461fd3a640 1 -- 172.21.15.3:0/3768735073 wait complete. 2024-03-25T18:35:26.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:26.213+0000 7f461fd3a640 1 librados: shutdown 2024-03-25T18:35:26.225 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:33.031 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:33.143 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.142+0000 7fca10ac1640 10 monclient: get_monmap_and_config 2024-03-25T18:35:33.144 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.142+0000 7fca10ac1640 10 monclient: build_initial_monmap 2024-03-25T18:35:33.144 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.142+0000 7fca10ac1640 10 monclient: monmap: 2024-03-25T18:35:33.144 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:35:33.144 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:33.144 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:35:33.144022+0000 2024-03-25T18:35:33.144 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:35:33.144022+0000 2024-03-25T18:35:33.144 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:33.144 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:33.144 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:35:33.144 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:35:33.144 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:35:33.144 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:33.145 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.143+0000 7fca10ac1640 1 Processor -- start 2024-03-25T18:35:33.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.143+0000 7fca10ac1640 1 -- start start 2024-03-25T18:35:33.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.143+0000 7fca10ac1640 10 monclient: init 2024-03-25T18:35:33.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.144+0000 7fca10ac1640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:33.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.144+0000 7fca10ac1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:35:33.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.144+0000 7fca10ac1640 10 monclient(hunting): picked mon.noname-c con 0x7fca0c0b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:35:33.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.144+0000 7fca10ac1640 10 monclient(hunting): picked mon.noname-b con 0x7fca0c0b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:35:33.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.144+0000 7fca10ac1640 10 monclient(hunting): picked mon.noname-a con 0x7fca0c13c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:35:33.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.144+0000 7fca10ac1640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c0ab0e0 con 0x7fca0c13c480 2024-03-25T18:35:33.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.144+0000 7fca10ac1640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c0ab3c0 con 0x7fca0c0b4ff0 2024-03-25T18:35:33.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.144+0000 7fca10ac1640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c0a9f30 con 0x7fca0c0b4670 2024-03-25T18:35:33.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.144+0000 7fca10ac1640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:33.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.144+0000 7fca10ac1640 10 monclient(hunting): authenticate will time out at 7987.404977s 2024-03-25T18:35:33.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.152+0000 7fca09d9b640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fca0c0b4670 0x7fca0c1425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:55544/0 (socket says 172.21.15.3:55544) 2024-03-25T18:35:33.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.152+0000 7fca09d9b640 1 -- 172.21.15.3:0/4137232168 learned_addr learned my addr 172.21.15.3:0/4137232168 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:35:33.171 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.170+0000 7fca0959a640 1 -- 172.21.15.3:0/4137232168 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 274222766 0 0) 0x7fca0c0a9f30 con 0x7fca0c0b4670 2024-03-25T18:35:33.171 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.170+0000 7fca0959a640 10 monclient(hunting): _negotiate 2024-03-25T18:35:33.171 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.170+0000 7fca0959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:33.171 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.170+0000 7fca0959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:33.171 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.170+0000 7fca0959a640 10 monclient(hunting): my global_id is 4655 2024-03-25T18:35:33.171 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.170+0000 7fca0959a640 1 -- 172.21.15.3:0/4137232168 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9f4003520 con 0x7fca0c0b4670 2024-03-25T18:35:33.173 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.172+0000 7fca0959a640 1 -- 172.21.15.3:0/4137232168 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3456756160 0 0) 0x7fca0c0ab0e0 con 0x7fca0c13c480 2024-03-25T18:35:33.173 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.172+0000 7fca0959a640 10 monclient(hunting): _negotiate 2024-03-25T18:35:33.173 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.172+0000 7fca0959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:33.173 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.172+0000 7fca0959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:33.173 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.172+0000 7fca0959a640 10 monclient(hunting): my global_id is 4704 2024-03-25T18:35:33.173 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.172+0000 7fca0959a640 1 -- 172.21.15.3:0/4137232168 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca0c0a9f30 con 0x7fca0c13c480 2024-03-25T18:35:33.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.176+0000 7fca0959a640 1 -- 172.21.15.3:0/4137232168 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2549500498 0 0) 0x7fca0c0ab3c0 con 0x7fca0c0b4ff0 2024-03-25T18:35:33.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.176+0000 7fca0959a640 10 monclient(hunting): _negotiate 2024-03-25T18:35:33.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.176+0000 7fca0959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:33.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.176+0000 7fca0959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:33.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.176+0000 7fca0959a640 10 monclient(hunting): my global_id is 4648 2024-03-25T18:35:33.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.176+0000 7fca0959a640 1 -- 172.21.15.3:0/4137232168 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca0c0ab0e0 con 0x7fca0c0b4ff0 2024-03-25T18:35:33.187 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.185+0000 7fca0959a640 1 -- 172.21.15.3:0/4137232168 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3799627583 0 0) 0x7fc9f4003520 con 0x7fca0c0b4670 2024-03-25T18:35:33.187 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.185+0000 7fca0959a640 1 -- 172.21.15.3:0/4137232168 >> v1:172.21.15.16:6789/0 conn(0x7fca0c0b4ff0 legacy=0x7fca0c13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:33.187 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.185+0000 7fca0959a640 1 -- 172.21.15.3:0/4137232168 >> v1:172.21.15.3:6789/0 conn(0x7fca0c13c480 legacy=0x7fca0c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:33.187 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.185+0000 7fca0959a640 10 monclient: _finish_hunting 0 2024-03-25T18:35:33.187 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.185+0000 7fca0959a640 1 monclient: found mon.noname-b 2024-03-25T18:35:33.187 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.185+0000 7fca0959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:33.187 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.185+0000 7fca0959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-25T18:35:33.187 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.185+0000 7fca0959a640 1 -- 172.21.15.3:0/4137232168 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0c0a9b70 con 0x7fca0c0b4670 2024-03-25T18:35:33.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.185+0000 7fca0959a640 10 monclient: _finish_auth 0 2024-03-25T18:35:33.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.185+0000 7fca0959a640 10 monclient: _check_auth_tickets 2024-03-25T18:35:33.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.185+0000 7fca0959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:33.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.185+0000 7fca0959a640 1 -- 172.21.15.3:0/4137232168 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fca00001690 con 0x7fca0c0b4670 2024-03-25T18:35:33.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.185+0000 7fca10ac1640 5 monclient: authenticate success, global_id 4655 2024-03-25T18:35:33.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.185+0000 7fca10ac1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:33.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.185+0000 7fca0959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:33.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.185+0000 7fca0959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:35:33.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.185+0000 7fca0959a640 10 monclient: dump: 2024-03-25T18:35:33.188 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:33.188 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:33.188 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:33.188 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:33.188 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:33.188 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:33.189 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:33.189 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:33.189 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:33.189 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:33.189 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.186+0000 7fca10ac1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:33.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.203+0000 7fca0959a640 1 -- 172.21.15.3:0/4137232168 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca000017f0 con 0x7fca0c0b4670 2024-03-25T18:35:33.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.203+0000 7fca0959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:33.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.203+0000 7fca0959a640 1 -- 172.21.15.3:0/4137232168 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fca00001c00 con 0x7fca0c0b4670 2024-03-25T18:35:33.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.203+0000 7fca10ac1640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:33.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.203+0000 7fca10ac1640 4 set_mon_vals no callback set 2024-03-25T18:35:33.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.203+0000 7fca0959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:33.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.203+0000 7fca0959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:33.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.203+0000 7fca0959a640 10 monclient: dump: 2024-03-25T18:35:33.205 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:33.205 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:33.205 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:33.205 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:33.205 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:33.205 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:33.205 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:33.205 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:33.206 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:33.206 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:33.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.204+0000 7fca10ac1640 10 monclient: shutdown 2024-03-25T18:35:33.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.204+0000 7fca10ac1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:33.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.204+0000 7fca10ac1640 1 -- 172.21.15.3:0/4137232168 >> v1:172.21.15.3:6790/0 conn(0x7fca0c0b4670 legacy=0x7fca0c1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:33.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.204+0000 7fca10ac1640 1 -- 172.21.15.3:0/4137232168 shutdown_connections 2024-03-25T18:35:33.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.204+0000 7fca10ac1640 1 -- 172.21.15.3:0/4137232168 >> 172.21.15.3:0/4137232168 conn(0x7fca0c0617a0 msgr2=0x7fca0c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:33.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.204+0000 7fca10ac1640 1 -- 172.21.15.3:0/4137232168 shutdown_connections 2024-03-25T18:35:33.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.204+0000 7fca10ac1640 1 -- 172.21.15.3:0/4137232168 wait complete. 2024-03-25T18:35:33.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.205+0000 7fca10ac1640 10 monclient: build_initial_monmap 2024-03-25T18:35:33.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.205+0000 7fca10ac1640 10 monclient: monmap: 2024-03-25T18:35:33.208 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:35:33.208 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:33.208 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:35:33.208 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:35:33.208 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:33.209 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:35:33.209 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:35:33.209 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:35:33.209 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:35:33.209 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:33.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.206+0000 7fca10ac1640 1 librados: starting msgr at 2024-03-25T18:35:33.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.206+0000 7fca10ac1640 1 librados: starting objecter 2024-03-25T18:35:33.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.206+0000 7fca10ac1640 1 Processor -- start 2024-03-25T18:35:33.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.206+0000 7fca10ac1640 1 -- start start 2024-03-25T18:35:33.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.206+0000 7fca10ac1640 1 librados: setting wanted keys 2024-03-25T18:35:33.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.206+0000 7fca10ac1640 1 librados: calling monclient init 2024-03-25T18:35:33.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.206+0000 7fca10ac1640 10 monclient: init 2024-03-25T18:35:33.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.206+0000 7fca10ac1640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:33.210 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.206+0000 7fca10ac1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:35:33.210 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.206+0000 7fca10ac1640 10 monclient(hunting): picked mon.noname-b con 0x7fca0c0b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:35:33.210 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.206+0000 7fca10ac1640 10 monclient(hunting): picked mon.noname-a con 0x7fca0c0b4ff0 addr v1:172.21.15.3:6789/0 2024-03-25T18:35:33.210 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.206+0000 7fca10ac1640 10 monclient(hunting): picked mon.noname-c con 0x7fca0c13c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:35:33.210 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.206+0000 7fca10ac1640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c13b120 con 0x7fca0c0b4ff0 2024-03-25T18:35:33.210 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.206+0000 7fca10ac1640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c0b1990 con 0x7fca0c0b4670 2024-03-25T18:35:33.210 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.206+0000 7fca10ac1640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c0b0130 con 0x7fca0c13c480 2024-03-25T18:35:33.210 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.206+0000 7fca10ac1640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:33.210 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.206+0000 7fca10ac1640 10 monclient(hunting): authenticate will time out at 7987.467164s 2024-03-25T18:35:33.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.212+0000 7fca0a59c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fca0c0b4670 0x7fca0c0b0e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:54882/0 (socket says 172.21.15.3:54882) 2024-03-25T18:35:33.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.212+0000 7fca0a59c640 1 -- 172.21.15.3:0/2212938681 learned_addr learned my addr 172.21.15.3:0/2212938681 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:35:33.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.213+0000 7fc9faffd640 10 client.?.objecter ms_handle_connect 0x7fca0c0b4670 2024-03-25T18:35:33.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.213+0000 7fc9faffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:33.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.214+0000 7fc9faffd640 10 client.?.objecter ms_handle_connect 0x7fca0c13c480 2024-03-25T18:35:33.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.214+0000 7fc9faffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:33.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.220+0000 7fc9faffd640 1 -- 172.21.15.3:0/2212938681 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 40526224 0 0) 0x7fca0c0b1990 con 0x7fca0c0b4670 2024-03-25T18:35:33.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.220+0000 7fc9faffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:33.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.220+0000 7fc9faffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:33.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.220+0000 7fc9faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:33.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.220+0000 7fc9faffd640 10 monclient(hunting): my global_id is 4651 2024-03-25T18:35:33.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.220+0000 7fc9faffd640 1 -- 172.21.15.3:0/2212938681 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9ec004eb0 con 0x7fca0c0b4670 2024-03-25T18:35:33.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.222+0000 7fc9faffd640 1 -- 172.21.15.3:0/2212938681 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2380290023 0 0) 0x7fca0c0b0130 con 0x7fca0c13c480 2024-03-25T18:35:33.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.222+0000 7fc9faffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:33.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.222+0000 7fc9faffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:33.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.222+0000 7fc9faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:33.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.222+0000 7fc9faffd640 10 monclient(hunting): my global_id is 4658 2024-03-25T18:35:33.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.222+0000 7fc9faffd640 1 -- 172.21.15.3:0/2212938681 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca0c0b1990 con 0x7fca0c13c480 2024-03-25T18:35:33.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.232+0000 7fc9faffd640 1 -- 172.21.15.3:0/2212938681 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2952325207 0 0) 0x7fc9ec004eb0 con 0x7fca0c0b4670 2024-03-25T18:35:33.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.232+0000 7fc9faffd640 1 -- 172.21.15.3:0/2212938681 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca0c0b0130 con 0x7fca0c0b4670 2024-03-25T18:35:33.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.232+0000 7fc9faffd640 1 -- 172.21.15.3:0/2212938681 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc9fc001a10 con 0x7fca0c0b4670 2024-03-25T18:35:33.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.232+0000 7fc9faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:33.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.232+0000 7fc9faffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:35:33.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.232+0000 7fc9faffd640 10 monclient(hunting): dump: 2024-03-25T18:35:33.234 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:33.234 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:33.234 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:33.234 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:33.234 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:33.235 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:33.235 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:33.235 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:33.235 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:33.235 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:33.235 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.232+0000 7fc9faffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:33.237 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.236+0000 7fc9faffd640 10 client.?.objecter ms_handle_connect 0x7fca0c0b4ff0 2024-03-25T18:35:33.237 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.236+0000 7fc9faffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:33.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.247+0000 7fc9faffd640 1 -- 172.21.15.3:0/2212938681 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1807463963 0 0) 0x7fca0c0b0130 con 0x7fca0c0b4670 2024-03-25T18:35:33.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.247+0000 7fc9faffd640 1 -- 172.21.15.3:0/2212938681 >> v1:172.21.15.3:6790/0 conn(0x7fca0c13c480 legacy=0x7fca0c0aeaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:33.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.247+0000 7fc9faffd640 1 -- 172.21.15.3:0/2212938681 >> v1:172.21.15.3:6789/0 conn(0x7fca0c0b4ff0 legacy=0x7fca0c0ae390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:33.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.247+0000 7fc9faffd640 10 monclient: _finish_hunting 0 2024-03-25T18:35:33.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.247+0000 7fc9faffd640 1 monclient: found mon.b 2024-03-25T18:35:33.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.247+0000 7fc9faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:33.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.247+0000 7fc9faffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:35:33.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.247+0000 7fc9faffd640 1 -- 172.21.15.3:0/2212938681 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0c0b0410 con 0x7fca0c0b4670 2024-03-25T18:35:33.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.247+0000 7fc9faffd640 10 monclient: _finish_auth 0 2024-03-25T18:35:33.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.247+0000 7fc9faffd640 10 monclient: _check_auth_tickets 2024-03-25T18:35:33.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.247+0000 7fc9faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:33.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.247+0000 7fca10ac1640 5 monclient: authenticate success, global_id 4651 2024-03-25T18:35:33.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.247+0000 7fca10ac1640 10 monclient: _renew_subs 2024-03-25T18:35:33.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.247+0000 7fca10ac1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:35:33.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.247+0000 7fca10ac1640 1 -- 172.21.15.3:0/2212938681 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca0c0b06f0 con 0x7fca0c0b4670 2024-03-25T18:35:33.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.247+0000 7fca10ac1640 10 client.4651.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:33.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.247+0000 7fca10ac1640 10 monclient: _renew_subs 2024-03-25T18:35:33.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.247+0000 7fca10ac1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:35:33.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.247+0000 7fca10ac1640 1 -- 172.21.15.3:0/2212938681 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca0c1e35a0 con 0x7fca0c0b4670 2024-03-25T18:35:33.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.247+0000 7fca10ac1640 1 librados: init done 2024-03-25T18:35:33.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.248+0000 7fca10ac1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:33.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.248+0000 7fca10ac1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:33.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.248+0000 7fca10ac1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:35:33.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.248+0000 7fca10ac1640 1 -- 172.21.15.3:0/2212938681 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca0c06dbd0 con 0x7fca0c0b4670 2024-03-25T18:35:33.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.547+0000 7fc9faffd640 1 -- 172.21.15.3:0/2212938681 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9fc001c70 con 0x7fca0c0b4670 2024-03-25T18:35:33.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.547+0000 7fc9faffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:33.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.547+0000 7fc9faffd640 1 -- 172.21.15.3:0/2212938681 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fc9fc003a60 con 0x7fca0c0b4670 2024-03-25T18:35:33.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.547+0000 7fc9faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:33.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.547+0000 7fc9faffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:33.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.547+0000 7fc9faffd640 10 monclient: dump: 2024-03-25T18:35:33.549 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:33.549 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:33.549 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:33.549 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:33.549 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:33.549 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:33.550 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:33.550 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:33.550 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:33.550 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:33.550 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.547+0000 7fca08d99640 4 set_mon_vals no callback set 2024-03-25T18:35:33.550 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.548+0000 7fc9faffd640 1 -- 172.21.15.3:0/2212938681 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7fc9fc003c80 con 0x7fca0c0b4670 2024-03-25T18:35:33.550 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.549+0000 7fca09d9b640 10 monclient: get_auth_request con 0x7fc9ec06dd10 auth_method 0 2024-03-25T18:35:33.551 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.549+0000 7fc9faffd640 10 client.4651.objecter ms_handle_connect 0x7fc9ec06dd10 2024-03-25T18:35:33.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.553+0000 7fc9faffd640 1 -- 172.21.15.3:0/2212938681 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2800263165 0 0) 0x7fc9fc054860 con 0x7fca0c0b4670 2024-03-25T18:35:33.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.553+0000 7fc9faffd640 10 client.4651.objecter ms_dispatch 0x7fca0c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:35:33.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.553+0000 7fc9faffd640 3 client.4651.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:35:33.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.553+0000 7fc9faffd640 3 client.4651.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:35:33.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.554+0000 7fc9faffd640 20 client.4651.objecter dump_active .. 0 homeless 2024-03-25T18:35:33.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.554+0000 7fc9faffd640 1 -- 172.21.15.3:0/2212938681 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc9fc050370 con 0x7fca0c0b4670 2024-03-25T18:35:33.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.554+0000 7fc9faffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:33.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.554+0000 7fc9faffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:33.784 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.783+0000 7fca10ac1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:33.784 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.783+0000 7fca10ac1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:33.784 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.783+0000 7fca10ac1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:35:33.784 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.783+0000 7fca10ac1640 1 -- 172.21.15.3:0/2212938681 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca0c0688c0 con 0x7fca0c0b4670 2024-03-25T18:35:33.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.870+0000 7fc9faffd640 1 -- 172.21.15.3:0/2212938681 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 716832884) 0x7fc9fc0522a0 con 0x7fca0c0b4670 2024-03-25T18:35:33.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.870+0000 7fc9faffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:33.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.870+0000 7fc9faffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:33.872 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:35:33.872 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":11,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:35:08.249841+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:35:08.249841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:33.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.872+0000 7fca10ac1640 10 librados: watch_flush enter 2024-03-25T18:35:33.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.872+0000 7fca10ac1640 10 librados: watch_flush exit 2024-03-25T18:35:33.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.872+0000 7fca10ac1640 20 client.4651.objecter shutdown clearing up homeless session... 2024-03-25T18:35:33.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.872+0000 7fca10ac1640 10 client.4651.objecter successfully canceled tick 2024-03-25T18:35:33.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.872+0000 7fca10ac1640 1 -- 172.21.15.3:0/2212938681 >> v1:172.21.15.3:6800/4195140966 conn(0x7fc9ec06dd10 legacy=0x7fc9ec0701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:33.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.872+0000 7fca10ac1640 10 monclient: shutdown 2024-03-25T18:35:33.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.872+0000 7fca10ac1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:33.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.872+0000 7fca10ac1640 1 -- 172.21.15.3:0/2212938681 >> v1:172.21.15.16:6789/0 conn(0x7fca0c0b4670 legacy=0x7fca0c0b0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:33.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.872+0000 7fca10ac1640 1 -- 172.21.15.3:0/2212938681 shutdown_connections 2024-03-25T18:35:33.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.872+0000 7fca10ac1640 1 -- 172.21.15.3:0/2212938681 >> 172.21.15.3:0/2212938681 conn(0x7fca0c0617a0 msgr2=0x7fca0c145670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:33.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.873+0000 7fca10ac1640 1 -- 172.21.15.3:0/2212938681 shutdown_connections 2024-03-25T18:35:33.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.873+0000 7fca10ac1640 1 -- 172.21.15.3:0/2212938681 wait complete. 2024-03-25T18:35:33.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:33.873+0000 7fca10ac1640 1 librados: shutdown 2024-03-25T18:35:33.884 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:40.692 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:40.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.803+0000 7f50b0f90640 10 monclient: get_monmap_and_config 2024-03-25T18:35:40.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.803+0000 7f50b0f90640 10 monclient: build_initial_monmap 2024-03-25T18:35:40.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.803+0000 7f50b0f90640 10 monclient: monmap: 2024-03-25T18:35:40.805 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:35:40.805 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:40.805 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:35:40.805031+0000 2024-03-25T18:35:40.805 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:35:40.805031+0000 2024-03-25T18:35:40.805 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:40.805 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:40.805 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:35:40.805 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:35:40.806 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:35:40.806 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:40.806 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.804+0000 7f50b0f90640 1 Processor -- start 2024-03-25T18:35:40.806 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.804+0000 7f50b0f90640 1 -- start start 2024-03-25T18:35:40.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.804+0000 7f50b0f90640 10 monclient: init 2024-03-25T18:35:40.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.805+0000 7f50b0f90640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:40.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.805+0000 7f50b0f90640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:35:40.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.805+0000 7f50b0f90640 10 monclient(hunting): picked mon.noname-c con 0x7f50ac0b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:35:40.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.805+0000 7f50b0f90640 10 monclient(hunting): picked mon.noname-b con 0x7f50ac0b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:35:40.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.805+0000 7f50b0f90640 10 monclient(hunting): picked mon.noname-a con 0x7f50ac13c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:35:40.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.805+0000 7f50b0f90640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50ac0ab0e0 con 0x7f50ac13c480 2024-03-25T18:35:40.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.805+0000 7f50b0f90640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50ac0ab3c0 con 0x7f50ac0b4ff0 2024-03-25T18:35:40.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.805+0000 7f50b0f90640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50ac0a9f30 con 0x7f50ac0b4670 2024-03-25T18:35:40.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.805+0000 7f50b0f90640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:40.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.805+0000 7f50b0f90640 10 monclient(hunting): authenticate will time out at 7995.065949s 2024-03-25T18:35:40.814 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.813+0000 7f50aa59c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f50ac0b4ff0 0x7f50ac13bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:37726/0 (socket says 172.21.15.3:37726) 2024-03-25T18:35:40.814 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.813+0000 7f50aa59c640 1 -- 172.21.15.3:0/1253628934 learned_addr learned my addr 172.21.15.3:0/1253628934 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:35:40.826 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.825+0000 7f50a9d9b640 1 -- 172.21.15.3:0/1253628934 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199994013 0 0) 0x7f50ac0ab3c0 con 0x7f50ac0b4ff0 2024-03-25T18:35:40.826 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.825+0000 7f50a9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:35:40.826 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.825+0000 7f50a9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:40.826 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.825+0000 7f50a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:40.826 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.825+0000 7f50a9d9b640 10 monclient(hunting): my global_id is 4654 2024-03-25T18:35:40.826 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.825+0000 7f50a9d9b640 1 -- 172.21.15.3:0/1253628934 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5098003520 con 0x7f50ac0b4ff0 2024-03-25T18:35:40.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.831+0000 7f50a9d9b640 1 -- 172.21.15.3:0/1253628934 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1413486034 0 0) 0x7f5098003520 con 0x7f50ac0b4ff0 2024-03-25T18:35:40.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.832+0000 7f50a9d9b640 1 -- 172.21.15.3:0/1253628934 >> v1:172.21.15.3:6790/0 conn(0x7f50ac0b4670 legacy=0x7f50ac1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:35:40.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.832+0000 7f50a9d9b640 1 -- 172.21.15.3:0/1253628934 >> v1:172.21.15.3:6789/0 conn(0x7f50ac13c480 legacy=0x7f50ac14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:40.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.832+0000 7f50a9d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:35:40.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.832+0000 7f50a9d9b640 1 monclient: found mon.noname-c 2024-03-25T18:35:40.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.832+0000 7f50a9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:40.834 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.832+0000 7f50a9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:35:40.834 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.832+0000 7f50a9d9b640 1 -- 172.21.15.3:0/1253628934 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ac0a9b70 con 0x7f50ac0b4ff0 2024-03-25T18:35:40.834 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.832+0000 7f50a9d9b640 10 monclient: _finish_auth 0 2024-03-25T18:35:40.834 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.832+0000 7f50a9d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:35:40.834 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.832+0000 7f50a9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:40.834 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.832+0000 7f50a9d9b640 1 -- 172.21.15.3:0/1253628934 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f509c001690 con 0x7f50ac0b4ff0 2024-03-25T18:35:40.834 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.832+0000 7f50b0f90640 5 monclient: authenticate success, global_id 4654 2024-03-25T18:35:40.834 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.832+0000 7f50a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:40.834 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.832+0000 7f50a9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:40.834 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.832+0000 7f50a9d9b640 10 monclient: dump: 2024-03-25T18:35:40.834 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:40.834 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:40.834 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:40.834 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:40.834 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:40.835 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:40.835 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:40.835 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:40.835 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:40.835 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:40.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.832+0000 7f50b0f90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:40.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.841+0000 7f50a9d9b640 1 -- 172.21.15.3:0/1253628934 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f509c001860 con 0x7f50ac0b4ff0 2024-03-25T18:35:40.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.841+0000 7f50a9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:40.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.841+0000 7f50a9d9b640 1 -- 172.21.15.3:0/1253628934 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f509c001c70 con 0x7f50ac0b4ff0 2024-03-25T18:35:40.843 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.841+0000 7f50b0f90640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:40.843 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.841+0000 7f50b0f90640 4 set_mon_vals no callback set 2024-03-25T18:35:40.843 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.841+0000 7f50a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:40.843 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.841+0000 7f50a9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:40.843 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.841+0000 7f50a9d9b640 10 monclient: dump: 2024-03-25T18:35:40.843 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:40.843 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:40.843 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:40.843 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:40.843 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:40.843 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:40.843 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:40.843 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:40.843 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:40.843 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:40.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.842+0000 7f50b0f90640 10 monclient: shutdown 2024-03-25T18:35:40.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.842+0000 7f50b0f90640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:40.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.842+0000 7f50b0f90640 1 -- 172.21.15.3:0/1253628934 >> v1:172.21.15.16:6789/0 conn(0x7f50ac0b4ff0 legacy=0x7f50ac13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:40.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.842+0000 7f50b0f90640 1 -- 172.21.15.3:0/1253628934 shutdown_connections 2024-03-25T18:35:40.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.842+0000 7f50b0f90640 1 -- 172.21.15.3:0/1253628934 >> 172.21.15.3:0/1253628934 conn(0x7f50ac0617a0 msgr2=0x7f50ac061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:40.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.842+0000 7f50b0f90640 1 -- 172.21.15.3:0/1253628934 shutdown_connections 2024-03-25T18:35:40.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.842+0000 7f50b0f90640 1 -- 172.21.15.3:0/1253628934 wait complete. 2024-03-25T18:35:40.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.843+0000 7f50b0f90640 10 monclient: build_initial_monmap 2024-03-25T18:35:40.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.843+0000 7f50b0f90640 10 monclient: monmap: 2024-03-25T18:35:40.845 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:35:40.846 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:40.846 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:35:40.846 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:35:40.846 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:40.846 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:35:40.846 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:35:40.846 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:35:40.846 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:35:40.846 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:40.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.844+0000 7f50b0f90640 1 librados: starting msgr at 2024-03-25T18:35:40.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.844+0000 7f50b0f90640 1 librados: starting objecter 2024-03-25T18:35:40.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.844+0000 7f50b0f90640 1 Processor -- start 2024-03-25T18:35:40.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.844+0000 7f50b0f90640 1 -- start start 2024-03-25T18:35:40.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.844+0000 7f50b0f90640 1 librados: setting wanted keys 2024-03-25T18:35:40.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.844+0000 7f50b0f90640 1 librados: calling monclient init 2024-03-25T18:35:40.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.844+0000 7f50b0f90640 10 monclient: init 2024-03-25T18:35:40.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.845+0000 7f50b0f90640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:40.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.845+0000 7f50b0f90640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:35:40.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.845+0000 7f50b0f90640 10 monclient(hunting): picked mon.noname-c con 0x7f50ac0b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:35:40.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.845+0000 7f50b0f90640 10 monclient(hunting): picked mon.noname-b con 0x7f50ac0b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:35:40.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.845+0000 7f50b0f90640 10 monclient(hunting): picked mon.noname-a con 0x7f50ac13c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:35:40.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.845+0000 7f50b0f90640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50ac06e640 con 0x7f50ac13c480 2024-03-25T18:35:40.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.845+0000 7f50b0f90640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50ac06e890 con 0x7f50ac0b4ff0 2024-03-25T18:35:40.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.845+0000 7f50b0f90640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50ac06eb70 con 0x7f50ac0b4670 2024-03-25T18:35:40.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.845+0000 7f50b0f90640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:40.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.845+0000 7f50b0f90640 10 monclient(hunting): authenticate will time out at 7995.106179s 2024-03-25T18:35:40.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.849+0000 7f50a1d9b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f50ac0b4ff0 0x7f50ac071710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:37762/0 (socket says 172.21.15.3:37762) 2024-03-25T18:35:40.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.849+0000 7f50a1d9b640 1 -- 172.21.15.3:0/2789256538 learned_addr learned my addr 172.21.15.3:0/2789256538 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:35:40.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.856+0000 7f50a37fe640 10 client.?.objecter ms_handle_connect 0x7f50ac0b4ff0 2024-03-25T18:35:40.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.856+0000 7f50a37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:40.860 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.859+0000 7f50a37fe640 10 client.?.objecter ms_handle_connect 0x7f50ac13c480 2024-03-25T18:35:40.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.859+0000 7f50a37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:40.863 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.862+0000 7f50a37fe640 10 client.?.objecter ms_handle_connect 0x7f50ac0b4670 2024-03-25T18:35:40.863 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.862+0000 7f50a37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:40.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.871+0000 7f50a37fe640 1 -- 172.21.15.3:0/2789256538 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1427037738 0 0) 0x7f50ac06e640 con 0x7f50ac13c480 2024-03-25T18:35:40.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.871+0000 7f50a37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:35:40.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.871+0000 7f50a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:40.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.871+0000 7f50a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:40.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.871+0000 7f50a37fe640 10 monclient(hunting): my global_id is 4713 2024-03-25T18:35:40.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.871+0000 7f50a37fe640 1 -- 172.21.15.3:0/2789256538 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f508c004eb0 con 0x7f50ac13c480 2024-03-25T18:35:40.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.887+0000 7f50a37fe640 1 -- 172.21.15.3:0/2789256538 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1974370850 0 0) 0x7f508c004eb0 con 0x7f50ac13c480 2024-03-25T18:35:40.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.887+0000 7f50a37fe640 1 -- 172.21.15.3:0/2789256538 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50ac06e640 con 0x7f50ac13c480 2024-03-25T18:35:40.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.887+0000 7f50a37fe640 1 -- 172.21.15.3:0/2789256538 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f50a4001b90 con 0x7f50ac13c480 2024-03-25T18:35:40.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.887+0000 7f50a37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:40.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.887+0000 7f50a37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:40.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.887+0000 7f50a37fe640 10 monclient(hunting): dump: 2024-03-25T18:35:40.889 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:40.889 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:40.889 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:40.889 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:40.889 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:40.889 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:40.889 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:40.889 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:40.889 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:40.889 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:40.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.887+0000 7f50a37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:40.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.909+0000 7f50a37fe640 1 -- 172.21.15.3:0/2789256538 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1690675417 0 0) 0x7f50ac06e640 con 0x7f50ac13c480 2024-03-25T18:35:40.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.909+0000 7f50a37fe640 1 -- 172.21.15.3:0/2789256538 >> v1:172.21.15.3:6790/0 conn(0x7f50ac0b4670 legacy=0x7f50ac1def20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:40.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.909+0000 7f50a37fe640 1 -- 172.21.15.3:0/2789256538 >> v1:172.21.15.16:6789/0 conn(0x7f50ac0b4ff0 legacy=0x7f50ac071710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:40.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.909+0000 7f50a37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:35:40.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.909+0000 7f50a37fe640 1 monclient: found mon.a 2024-03-25T18:35:40.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.909+0000 7f50a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:40.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.909+0000 7f50a37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:35:40.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.909+0000 7f50a37fe640 1 -- 172.21.15.3:0/2789256538 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ac146100 con 0x7f50ac13c480 2024-03-25T18:35:40.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.909+0000 7f50a37fe640 10 monclient: _finish_auth 0 2024-03-25T18:35:40.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.909+0000 7f50a37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:35:40.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.909+0000 7f50a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:40.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.909+0000 7f50b0f90640 5 monclient: authenticate success, global_id 4713 2024-03-25T18:35:40.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.909+0000 7f50b0f90640 10 monclient: _renew_subs 2024-03-25T18:35:40.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.909+0000 7f50b0f90640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:35:40.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.909+0000 7f50b0f90640 1 -- 172.21.15.3:0/2789256538 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50ac145d90 con 0x7f50ac13c480 2024-03-25T18:35:40.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.909+0000 7f50b0f90640 10 client.4713.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:40.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.909+0000 7f50b0f90640 10 monclient: _renew_subs 2024-03-25T18:35:40.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.909+0000 7f50b0f90640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:35:40.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.909+0000 7f50b0f90640 1 -- 172.21.15.3:0/2789256538 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50ac1458e0 con 0x7f50ac13c480 2024-03-25T18:35:40.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.909+0000 7f50b0f90640 1 librados: init done 2024-03-25T18:35:40.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.910+0000 7f50b0f90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:40.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.910+0000 7f50b0f90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:40.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.910+0000 7f50b0f90640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:35:40.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:40.910+0000 7f50b0f90640 1 -- 172.21.15.3:0/2789256538 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50ac06f090 con 0x7f50ac13c480 2024-03-25T18:35:41.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.220+0000 7f50a37fe640 1 -- 172.21.15.3:0/2789256538 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50a40020c0 con 0x7f50ac13c480 2024-03-25T18:35:41.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.220+0000 7f50a37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:41.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.220+0000 7f50a37fe640 1 -- 172.21.15.3:0/2789256538 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f50a4003990 con 0x7f50ac13c480 2024-03-25T18:35:41.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.220+0000 7f50a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:41.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.220+0000 7f50a37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:41.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.220+0000 7f50a37fe640 10 monclient: dump: 2024-03-25T18:35:41.222 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:41.222 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:41.222 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:41.222 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:41.222 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:41.222 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:41.222 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:41.222 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:41.222 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:41.223 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:41.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.220+0000 7f50a959a640 4 set_mon_vals no callback set 2024-03-25T18:35:41.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.221+0000 7f50a37fe640 1 -- 172.21.15.3:0/2789256538 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f50a4003b70 con 0x7f50ac13c480 2024-03-25T18:35:41.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.221+0000 7f50aa59c640 10 monclient: get_auth_request con 0x7f508c06dd00 auth_method 0 2024-03-25T18:35:41.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.222+0000 7f50a37fe640 10 client.4713.objecter ms_handle_connect 0x7f508c06dd00 2024-03-25T18:35:41.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.232+0000 7f50a37fe640 1 -- 172.21.15.3:0/2789256538 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2800263165 0 0) 0x7f50a4053710 con 0x7f50ac13c480 2024-03-25T18:35:41.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.232+0000 7f50a37fe640 10 client.4713.objecter ms_dispatch 0x7f50ac000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:35:41.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.232+0000 7f50a37fe640 3 client.4713.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:35:41.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.232+0000 7f50a37fe640 3 client.4713.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:35:41.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.232+0000 7f50a37fe640 20 client.4713.objecter dump_active .. 0 homeless 2024-03-25T18:35:41.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.232+0000 7f50a37fe640 1 -- 172.21.15.3:0/2789256538 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f50a404f050 con 0x7f50ac13c480 2024-03-25T18:35:41.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.232+0000 7f50a37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:41.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.232+0000 7f50a37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:41.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.487+0000 7f50b0f90640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:41.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.487+0000 7f50b0f90640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:41.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.487+0000 7f50b0f90640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:35:41.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.487+0000 7f50b0f90640 1 -- 172.21.15.3:0/2789256538 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50ac0688c0 con 0x7f50ac13c480 2024-03-25T18:35:41.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.514+0000 7f50a37fe640 1 -- 172.21.15.3:0/2789256538 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 716832884) 0x7f50a4051150 con 0x7f50ac13c480 2024-03-25T18:35:41.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.514+0000 7f50a37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:41.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.514+0000 7f50a37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:41.516 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:35:41.516 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":11,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:35:08.249841+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:35:08.249841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:41.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.516+0000 7f50b0f90640 10 librados: watch_flush enter 2024-03-25T18:35:41.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.516+0000 7f50b0f90640 10 librados: watch_flush exit 2024-03-25T18:35:41.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.516+0000 7f50b0f90640 20 client.4713.objecter shutdown clearing up homeless session... 2024-03-25T18:35:41.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.516+0000 7f50b0f90640 10 client.4713.objecter successfully canceled tick 2024-03-25T18:35:41.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.516+0000 7f50b0f90640 1 -- 172.21.15.3:0/2789256538 >> v1:172.21.15.3:6800/4195140966 conn(0x7f508c06dd00 legacy=0x7f508c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:41.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.516+0000 7f50b0f90640 10 monclient: shutdown 2024-03-25T18:35:41.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.516+0000 7f50b0f90640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:41.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.516+0000 7f50b0f90640 1 -- 172.21.15.3:0/2789256538 >> v1:172.21.15.3:6789/0 conn(0x7f50ac13c480 legacy=0x7f50ac1e72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:41.519 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.516+0000 7f50b0f90640 1 -- 172.21.15.3:0/2789256538 shutdown_connections 2024-03-25T18:35:41.519 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.517+0000 7f50b0f90640 1 -- 172.21.15.3:0/2789256538 >> 172.21.15.3:0/2789256538 conn(0x7f50ac0617a0 msgr2=0x7f50ac1405c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:41.519 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.517+0000 7f50b0f90640 1 -- 172.21.15.3:0/2789256538 shutdown_connections 2024-03-25T18:35:41.519 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.517+0000 7f50b0f90640 1 -- 172.21.15.3:0/2789256538 wait complete. 2024-03-25T18:35:41.519 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:41.517+0000 7f50b0f90640 1 librados: shutdown 2024-03-25T18:35:41.531 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:43.491 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:10:57.220 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:43.498 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:10:57.227 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:43.529 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:10:57.258 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:43.549 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:10:57.278 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:43.562 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:10:57.292 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:43.573 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:10:57.265 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:43.577 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:10:57.269 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:43.591 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:10:57.321 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:43.619 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:10:57.311 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:48.337 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:48.451 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.450+0000 7fe55340e640 10 monclient: get_monmap_and_config 2024-03-25T18:35:48.452 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.450+0000 7fe55340e640 10 monclient: build_initial_monmap 2024-03-25T18:35:48.452 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.451+0000 7fe55340e640 10 monclient: monmap: 2024-03-25T18:35:48.452 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:35:48.452 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:48.452 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:35:48.452092+0000 2024-03-25T18:35:48.452 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:35:48.452092+0000 2024-03-25T18:35:48.452 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:48.452 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:48.452 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:35:48.452 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:35:48.452 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:35:48.452 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:48.453 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.451+0000 7fe55340e640 1 Processor -- start 2024-03-25T18:35:48.453 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.451+0000 7fe55340e640 1 -- start start 2024-03-25T18:35:48.453 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.451+0000 7fe55340e640 10 monclient: init 2024-03-25T18:35:48.453 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.452+0000 7fe55340e640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:48.453 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.452+0000 7fe55340e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:35:48.453 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.452+0000 7fe55340e640 10 monclient(hunting): picked mon.noname-c con 0x7fe54c0703b0 addr v1:172.21.15.16:6789/0 2024-03-25T18:35:48.454 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.452+0000 7fe55340e640 10 monclient(hunting): picked mon.noname-b con 0x7fe54c140ba0 addr v1:172.21.15.3:6790/0 2024-03-25T18:35:48.454 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.452+0000 7fe55340e640 10 monclient(hunting): picked mon.noname-a con 0x7fe54c0af6e0 addr v1:172.21.15.3:6789/0 2024-03-25T18:35:48.454 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.452+0000 7fe55340e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54c0723b0 con 0x7fe54c0af6e0 2024-03-25T18:35:48.454 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.452+0000 7fe55340e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54c141020 con 0x7fe54c0703b0 2024-03-25T18:35:48.454 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.452+0000 7fe55340e640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54c072600 con 0x7fe54c140ba0 2024-03-25T18:35:48.454 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.452+0000 7fe55340e640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:48.454 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.452+0000 7fe55340e640 10 monclient(hunting): authenticate will time out at 8002.712978s 2024-03-25T18:35:48.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.464+0000 7fe5511aa640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe54c0703b0 0x7fe54c070790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:41878/0 (socket says 172.21.15.3:41878) 2024-03-25T18:35:48.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.464+0000 7fe5511aa640 1 -- 172.21.15.3:0/1094231468 learned_addr learned my addr 172.21.15.3:0/1094231468 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:35:48.475 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.474+0000 7fe543fff640 1 -- 172.21.15.3:0/1094231468 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3969354409 0 0) 0x7fe54c141020 con 0x7fe54c0703b0 2024-03-25T18:35:48.475 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.474+0000 7fe543fff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:48.475 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.474+0000 7fe543fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:48.476 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.474+0000 7fe543fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:48.476 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.474+0000 7fe543fff640 10 monclient(hunting): my global_id is 4657 2024-03-25T18:35:48.476 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.474+0000 7fe543fff640 1 -- 172.21.15.3:0/1094231468 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe538003520 con 0x7fe54c0703b0 2024-03-25T18:35:48.483 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.482+0000 7fe543fff640 1 -- 172.21.15.3:0/1094231468 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104598873 0 0) 0x7fe54c0723b0 con 0x7fe54c0af6e0 2024-03-25T18:35:48.483 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.482+0000 7fe543fff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:48.483 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.482+0000 7fe543fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:48.483 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.482+0000 7fe543fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:48.483 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.482+0000 7fe543fff640 10 monclient(hunting): my global_id is 4716 2024-03-25T18:35:48.484 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.482+0000 7fe543fff640 1 -- 172.21.15.3:0/1094231468 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe54c141020 con 0x7fe54c0af6e0 2024-03-25T18:35:48.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.487+0000 7fe543fff640 1 -- 172.21.15.3:0/1094231468 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4244723922 0 0) 0x7fe538003520 con 0x7fe54c0703b0 2024-03-25T18:35:48.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.487+0000 7fe543fff640 1 -- 172.21.15.3:0/1094231468 >> v1:172.21.15.3:6790/0 conn(0x7fe54c140ba0 legacy=0x7fe54c139cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:48.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.487+0000 7fe543fff640 1 -- 172.21.15.3:0/1094231468 >> v1:172.21.15.3:6789/0 conn(0x7fe54c0af6e0 legacy=0x7fe54c14b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:48.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.487+0000 7fe543fff640 10 monclient: _finish_hunting 0 2024-03-25T18:35:48.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.487+0000 7fe543fff640 1 monclient: found mon.noname-c 2024-03-25T18:35:48.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.487+0000 7fe543fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:48.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.487+0000 7fe543fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:35:48.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.487+0000 7fe543fff640 1 -- 172.21.15.3:0/1094231468 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54c0728e0 con 0x7fe54c0703b0 2024-03-25T18:35:48.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.487+0000 7fe543fff640 10 monclient: _finish_auth 0 2024-03-25T18:35:48.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.487+0000 7fe543fff640 10 monclient: _check_auth_tickets 2024-03-25T18:35:48.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.487+0000 7fe543fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:48.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.487+0000 7fe543fff640 1 -- 172.21.15.3:0/1094231468 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fe548001690 con 0x7fe54c0703b0 2024-03-25T18:35:48.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.487+0000 7fe543fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:48.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.487+0000 7fe543fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:48.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.487+0000 7fe543fff640 10 monclient: dump: 2024-03-25T18:35:48.489 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:48.490 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:48.490 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:48.490 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:48.490 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:48.490 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:48.490 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:48.490 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:48.490 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:48.490 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:48.490 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.487+0000 7fe55340e640 5 monclient: authenticate success, global_id 4657 2024-03-25T18:35:48.490 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.487+0000 7fe55340e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:48.496 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.495+0000 7fe543fff640 1 -- 172.21.15.3:0/1094231468 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5480017f0 con 0x7fe54c0703b0 2024-03-25T18:35:48.496 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.495+0000 7fe543fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:48.496 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.495+0000 7fe543fff640 1 -- 172.21.15.3:0/1094231468 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fe548001b80 con 0x7fe54c0703b0 2024-03-25T18:35:48.496 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.495+0000 7fe55340e640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:48.496 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.495+0000 7fe55340e640 4 set_mon_vals no callback set 2024-03-25T18:35:48.497 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.495+0000 7fe543fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:48.497 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.495+0000 7fe543fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:48.497 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.495+0000 7fe543fff640 10 monclient: dump: 2024-03-25T18:35:48.497 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:48.497 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:48.497 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:48.497 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:48.497 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:48.497 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:48.497 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:48.497 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:48.497 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:48.497 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:48.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.496+0000 7fe55340e640 10 monclient: shutdown 2024-03-25T18:35:48.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.496+0000 7fe55340e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:48.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.496+0000 7fe55340e640 1 -- 172.21.15.3:0/1094231468 >> v1:172.21.15.16:6789/0 conn(0x7fe54c0703b0 legacy=0x7fe54c070790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:48.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.496+0000 7fe55340e640 1 -- 172.21.15.3:0/1094231468 shutdown_connections 2024-03-25T18:35:48.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.496+0000 7fe55340e640 1 -- 172.21.15.3:0/1094231468 >> 172.21.15.3:0/1094231468 conn(0x7fe54c0b3cd0 msgr2=0x7fe54c0b40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:48.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.496+0000 7fe55340e640 1 -- 172.21.15.3:0/1094231468 shutdown_connections 2024-03-25T18:35:48.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.496+0000 7fe55340e640 1 -- 172.21.15.3:0/1094231468 wait complete. 2024-03-25T18:35:48.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.497+0000 7fe55340e640 10 monclient: build_initial_monmap 2024-03-25T18:35:48.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.497+0000 7fe55340e640 10 monclient: monmap: 2024-03-25T18:35:48.499 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:35:48.499 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:48.499 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:35:48.499 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:35:48.500 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:48.500 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:35:48.500 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:35:48.500 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:35:48.500 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:35:48.500 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:48.500 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.497+0000 7fe55340e640 1 librados: starting msgr at 2024-03-25T18:35:48.500 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.497+0000 7fe55340e640 1 librados: starting objecter 2024-03-25T18:35:48.500 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.498+0000 7fe55340e640 1 Processor -- start 2024-03-25T18:35:48.500 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.498+0000 7fe55340e640 1 -- start start 2024-03-25T18:35:48.500 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.498+0000 7fe55340e640 1 librados: setting wanted keys 2024-03-25T18:35:48.500 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.498+0000 7fe55340e640 1 librados: calling monclient init 2024-03-25T18:35:48.500 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.498+0000 7fe55340e640 10 monclient: init 2024-03-25T18:35:48.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.498+0000 7fe55340e640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:48.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.498+0000 7fe55340e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:35:48.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.498+0000 7fe55340e640 10 monclient(hunting): picked mon.noname-b con 0x7fe54c0703b0 addr v1:172.21.15.16:6789/0 2024-03-25T18:35:48.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.498+0000 7fe55340e640 10 monclient(hunting): picked mon.noname-a con 0x7fe54c0af6e0 addr v1:172.21.15.3:6789/0 2024-03-25T18:35:48.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.498+0000 7fe55340e640 10 monclient(hunting): picked mon.noname-c con 0x7fe54c140ba0 addr v1:172.21.15.3:6790/0 2024-03-25T18:35:48.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.498+0000 7fe55340e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54c136a20 con 0x7fe54c0af6e0 2024-03-25T18:35:48.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.498+0000 7fe55340e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54c13d410 con 0x7fe54c0703b0 2024-03-25T18:35:48.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.498+0000 7fe55340e640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54c1d7400 con 0x7fe54c140ba0 2024-03-25T18:35:48.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.498+0000 7fe55340e640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:48.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.498+0000 7fe55340e640 10 monclient(hunting): authenticate will time out at 8002.759217s 2024-03-25T18:35:48.509 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.509+0000 7fe5511aa640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe54c0703b0 0x7fe54c1d7920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:41916/0 (socket says 172.21.15.3:41916) 2024-03-25T18:35:48.510 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.509+0000 7fe5511aa640 1 -- 172.21.15.3:0/859964676 learned_addr learned my addr 172.21.15.3:0/859964676 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:35:48.511 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.510+0000 7fe541ffb640 10 client.?.objecter ms_handle_connect 0x7fe54c0703b0 2024-03-25T18:35:48.511 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.510+0000 7fe541ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:48.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.514+0000 7fe541ffb640 1 -- 172.21.15.3:0/859964676 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2571849250 0 0) 0x7fe54c13d410 con 0x7fe54c0703b0 2024-03-25T18:35:48.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.514+0000 7fe541ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:35:48.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.514+0000 7fe541ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:48.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.514+0000 7fe541ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:48.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.514+0000 7fe541ffb640 10 monclient(hunting): my global_id is 4660 2024-03-25T18:35:48.516 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.514+0000 7fe541ffb640 1 -- 172.21.15.3:0/859964676 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe530004eb0 con 0x7fe54c0703b0 2024-03-25T18:35:48.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.522+0000 7fe541ffb640 10 client.?.objecter ms_handle_connect 0x7fe54c0af6e0 2024-03-25T18:35:48.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.522+0000 7fe541ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:48.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.524+0000 7fe541ffb640 10 client.?.objecter ms_handle_connect 0x7fe54c140ba0 2024-03-25T18:35:48.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.524+0000 7fe541ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:48.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.526+0000 7fe541ffb640 1 -- 172.21.15.3:0/859964676 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3080197531 0 0) 0x7fe530004eb0 con 0x7fe54c0703b0 2024-03-25T18:35:48.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.526+0000 7fe541ffb640 1 -- 172.21.15.3:0/859964676 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe54c13d410 con 0x7fe54c0703b0 2024-03-25T18:35:48.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.526+0000 7fe541ffb640 1 -- 172.21.15.3:0/859964676 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fe548001bc0 con 0x7fe54c0703b0 2024-03-25T18:35:48.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.526+0000 7fe541ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:48.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.526+0000 7fe541ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:35:48.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.526+0000 7fe541ffb640 10 monclient(hunting): dump: 2024-03-25T18:35:48.528 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:48.528 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:48.528 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:48.528 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:48.528 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:48.528 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:48.528 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:48.528 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:48.529 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:48.529 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:48.529 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.526+0000 7fe541ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:48.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.530+0000 7fe541ffb640 1 -- 172.21.15.3:0/859964676 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700537066 0 0) 0x7fe54c136a20 con 0x7fe54c0af6e0 2024-03-25T18:35:48.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.530+0000 7fe541ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:35:48.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.530+0000 7fe541ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:48.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.530+0000 7fe541ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:48.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.530+0000 7fe541ffb640 10 monclient(hunting): my global_id is 4719 2024-03-25T18:35:48.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.530+0000 7fe541ffb640 1 -- 172.21.15.3:0/859964676 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe530004eb0 con 0x7fe54c0af6e0 2024-03-25T18:35:48.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.536+0000 7fe541ffb640 1 -- 172.21.15.3:0/859964676 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2494515164 0 0) 0x7fe54c13d410 con 0x7fe54c0703b0 2024-03-25T18:35:48.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.536+0000 7fe541ffb640 1 -- 172.21.15.3:0/859964676 >> v1:172.21.15.3:6790/0 conn(0x7fe54c140ba0 legacy=0x7fe54c136310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:48.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.536+0000 7fe541ffb640 1 -- 172.21.15.3:0/859964676 >> v1:172.21.15.3:6789/0 conn(0x7fe54c0af6e0 legacy=0x7fe54c135c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:48.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.536+0000 7fe541ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:35:48.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.536+0000 7fe541ffb640 1 monclient: found mon.b 2024-03-25T18:35:48.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.536+0000 7fe541ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:48.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.536+0000 7fe541ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:35:48.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.536+0000 7fe541ffb640 1 -- 172.21.15.3:0/859964676 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54c13b090 con 0x7fe54c0703b0 2024-03-25T18:35:48.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.536+0000 7fe541ffb640 10 monclient: _finish_auth 0 2024-03-25T18:35:48.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.536+0000 7fe541ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:35:48.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.536+0000 7fe541ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:48.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.536+0000 7fe55340e640 5 monclient: authenticate success, global_id 4660 2024-03-25T18:35:48.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.536+0000 7fe55340e640 10 monclient: _renew_subs 2024-03-25T18:35:48.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.536+0000 7fe55340e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:35:48.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.536+0000 7fe55340e640 1 -- 172.21.15.3:0/859964676 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe54c13c480 con 0x7fe54c0703b0 2024-03-25T18:35:48.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.536+0000 7fe55340e640 10 client.4660.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:48.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.536+0000 7fe55340e640 10 monclient: _renew_subs 2024-03-25T18:35:48.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.536+0000 7fe55340e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:35:48.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.536+0000 7fe55340e640 1 -- 172.21.15.3:0/859964676 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe54c06de40 con 0x7fe54c0703b0 2024-03-25T18:35:48.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.536+0000 7fe55340e640 1 librados: init done 2024-03-25T18:35:48.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.537+0000 7fe55340e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:48.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.537+0000 7fe55340e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:48.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.537+0000 7fe55340e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:35:48.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.537+0000 7fe55340e640 1 -- 172.21.15.3:0/859964676 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe54c1d7590 con 0x7fe54c0703b0 2024-03-25T18:35:48.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.827+0000 7fe541ffb640 1 -- 172.21.15.3:0/859964676 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe548001270 con 0x7fe54c0703b0 2024-03-25T18:35:48.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.827+0000 7fe541ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:48.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.827+0000 7fe541ffb640 1 -- 172.21.15.3:0/859964676 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fe548004c30 con 0x7fe54c0703b0 2024-03-25T18:35:48.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.827+0000 7fe541ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:48.829 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.827+0000 7fe541ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:48.829 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.827+0000 7fe541ffb640 10 monclient: dump: 2024-03-25T18:35:48.829 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:48.829 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:48.829 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:48.829 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:48.829 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:48.829 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:48.829 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:48.829 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:48.829 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:48.829 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:48.829 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.827+0000 7fe5437fe640 4 set_mon_vals no callback set 2024-03-25T18:35:48.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.828+0000 7fe541ffb640 1 -- 172.21.15.3:0/859964676 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7fe54801ac80 con 0x7fe54c0703b0 2024-03-25T18:35:48.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.829+0000 7fe5509a9640 10 monclient: get_auth_request con 0x7fe53006de20 auth_method 0 2024-03-25T18:35:48.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.829+0000 7fe541ffb640 10 client.4660.objecter ms_handle_connect 0x7fe53006de20 2024-03-25T18:35:48.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.834+0000 7fe541ffb640 1 -- 172.21.15.3:0/859964676 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2800263165 0 0) 0x7fe5480548b0 con 0x7fe54c0703b0 2024-03-25T18:35:48.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.834+0000 7fe541ffb640 10 client.4660.objecter ms_dispatch 0x7fe54c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:35:48.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.834+0000 7fe541ffb640 3 client.4660.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:35:48.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.834+0000 7fe541ffb640 3 client.4660.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:35:48.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.834+0000 7fe541ffb640 20 client.4660.objecter dump_active .. 0 homeless 2024-03-25T18:35:48.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.834+0000 7fe541ffb640 1 -- 172.21.15.3:0/859964676 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe5480501f0 con 0x7fe54c0703b0 2024-03-25T18:35:48.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.834+0000 7fe541ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:48.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:48.834+0000 7fe541ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:49.088 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:49.087+0000 7fe55340e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:49.088 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:49.087+0000 7fe55340e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:49.089 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:49.087+0000 7fe55340e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:35:49.089 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:49.087+0000 7fe55340e640 1 -- 172.21.15.3:0/859964676 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe54c04f730 con 0x7fe54c0703b0 2024-03-25T18:35:49.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:49.108+0000 7fe541ffb640 1 -- 172.21.15.3:0/859964676 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 716832884) 0x7fe5480522f0 con 0x7fe54c0703b0 2024-03-25T18:35:49.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:49.108+0000 7fe541ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:49.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:49.108+0000 7fe541ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:49.110 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:35:49.110 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":11,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:35:08.249841+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:35:08.249841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:49.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:49.111+0000 7fe55340e640 10 librados: watch_flush enter 2024-03-25T18:35:49.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:49.111+0000 7fe55340e640 10 librados: watch_flush exit 2024-03-25T18:35:49.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:49.111+0000 7fe55340e640 20 client.4660.objecter shutdown clearing up homeless session... 2024-03-25T18:35:49.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:49.111+0000 7fe55340e640 10 client.4660.objecter successfully canceled tick 2024-03-25T18:35:49.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:49.111+0000 7fe55340e640 1 -- 172.21.15.3:0/859964676 >> v1:172.21.15.3:6800/4195140966 conn(0x7fe53006de20 legacy=0x7fe5300702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:49.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:49.111+0000 7fe55340e640 10 monclient: shutdown 2024-03-25T18:35:49.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:49.111+0000 7fe55340e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:49.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:49.111+0000 7fe55340e640 1 -- 172.21.15.3:0/859964676 >> v1:172.21.15.16:6789/0 conn(0x7fe54c0703b0 legacy=0x7fe54c1d7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:49.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:49.111+0000 7fe55340e640 1 -- 172.21.15.3:0/859964676 shutdown_connections 2024-03-25T18:35:49.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:49.111+0000 7fe55340e640 1 -- 172.21.15.3:0/859964676 >> 172.21.15.3:0/859964676 conn(0x7fe54c0b3cd0 msgr2=0x7fe54c0b1060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:49.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:49.111+0000 7fe55340e640 1 -- 172.21.15.3:0/859964676 shutdown_connections 2024-03-25T18:35:49.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:49.111+0000 7fe55340e640 1 -- 172.21.15.3:0/859964676 wait complete. 2024-03-25T18:35:49.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:49.111+0000 7fe55340e640 1 librados: shutdown 2024-03-25T18:35:49.125 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:55.931 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:56.034 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.033+0000 7fb117b25640 10 monclient: get_monmap_and_config 2024-03-25T18:35:56.034 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.033+0000 7fb117b25640 10 monclient: build_initial_monmap 2024-03-25T18:35:56.034 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.033+0000 7fb117b25640 10 monclient: monmap: 2024-03-25T18:35:56.034 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:35:56.035 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:56.035 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:35:56.034635+0000 2024-03-25T18:35:56.035 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:35:56.034635+0000 2024-03-25T18:35:56.035 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:56.035 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:56.035 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:35:56.035 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:35:56.035 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:35:56.035 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:56.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.034+0000 7fb117b25640 1 Processor -- start 2024-03-25T18:35:56.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.034+0000 7fb117b25640 1 -- start start 2024-03-25T18:35:56.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.034+0000 7fb117b25640 10 monclient: init 2024-03-25T18:35:56.037 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.034+0000 7fb117b25640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:56.037 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.034+0000 7fb117b25640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:35:56.037 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.035+0000 7fb117b25640 10 monclient(hunting): picked mon.noname-c con 0x7fb1100b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:35:56.037 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.035+0000 7fb117b25640 10 monclient(hunting): picked mon.noname-a con 0x7fb1100b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:35:56.037 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.035+0000 7fb117b25640 10 monclient(hunting): picked mon.noname-b con 0x7fb11013c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:35:56.037 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.035+0000 7fb117b25640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1100ab0e0 con 0x7fb1100b4670 2024-03-25T18:35:56.037 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.035+0000 7fb117b25640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1100ab3c0 con 0x7fb1100b4ff0 2024-03-25T18:35:56.037 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.035+0000 7fb117b25640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1100a9f30 con 0x7fb11013c480 2024-03-25T18:35:56.037 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.035+0000 7fb117b25640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:56.037 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.035+0000 7fb117b25640 10 monclient(hunting): authenticate will time out at 8010.295436s 2024-03-25T18:35:56.039 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.038+0000 7fb1150c0640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fb1100b4670 0x7fb1101425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:45406/0 (socket says 172.21.15.3:45406) 2024-03-25T18:35:56.039 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.038+0000 7fb1150c0640 1 -- 172.21.15.3:0/1849892653 learned_addr learned my addr 172.21.15.3:0/1849892653 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:35:56.054 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.053+0000 7fb1148bf640 1 -- 172.21.15.3:0/1849892653 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 789762536 0 0) 0x7fb1100ab0e0 con 0x7fb1100b4670 2024-03-25T18:35:56.054 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.053+0000 7fb1148bf640 10 monclient(hunting): _negotiate 2024-03-25T18:35:56.054 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.053+0000 7fb1148bf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:56.054 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.053+0000 7fb1148bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:56.054 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.053+0000 7fb1148bf640 10 monclient(hunting): my global_id is 4722 2024-03-25T18:35:56.054 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.053+0000 7fb1148bf640 1 -- 172.21.15.3:0/1849892653 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0fc003520 con 0x7fb1100b4670 2024-03-25T18:35:56.090 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.089+0000 7fb1148bf640 1 -- 172.21.15.3:0/1849892653 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2979614711 0 0) 0x7fb0fc003520 con 0x7fb1100b4670 2024-03-25T18:35:56.090 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.089+0000 7fb1148bf640 1 -- 172.21.15.3:0/1849892653 >> v1:172.21.15.3:6790/0 conn(0x7fb11013c480 legacy=0x7fb11014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:35:56.090 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.089+0000 7fb1148bf640 1 -- 172.21.15.3:0/1849892653 >> v1:172.21.15.16:6789/0 conn(0x7fb1100b4ff0 legacy=0x7fb11013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:56.090 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.089+0000 7fb1148bf640 10 monclient: _finish_hunting 0 2024-03-25T18:35:56.090 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.089+0000 7fb1148bf640 1 monclient: found mon.noname-a 2024-03-25T18:35:56.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.089+0000 7fb1148bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:56.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.089+0000 7fb1148bf640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:35:56.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.089+0000 7fb1148bf640 1 -- 172.21.15.3:0/1849892653 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1100a9b70 con 0x7fb1100b4670 2024-03-25T18:35:56.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.089+0000 7fb1148bf640 10 monclient: _finish_auth 0 2024-03-25T18:35:56.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.089+0000 7fb1148bf640 10 monclient: _check_auth_tickets 2024-03-25T18:35:56.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.089+0000 7fb1148bf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:56.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.089+0000 7fb1148bf640 1 -- 172.21.15.3:0/1849892653 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fb100001690 con 0x7fb1100b4670 2024-03-25T18:35:56.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.089+0000 7fb1148bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:56.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.089+0000 7fb1148bf640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:56.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.089+0000 7fb1148bf640 10 monclient: dump: 2024-03-25T18:35:56.091 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:56.091 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:56.091 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:56.091 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:56.092 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:56.092 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:56.092 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:56.092 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:56.092 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:56.092 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:56.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.089+0000 7fb117b25640 5 monclient: authenticate success, global_id 4722 2024-03-25T18:35:56.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.089+0000 7fb117b25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:56.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.106+0000 7fb1148bf640 1 -- 172.21.15.3:0/1849892653 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb100001860 con 0x7fb1100b4670 2024-03-25T18:35:56.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.106+0000 7fb1148bf640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:56.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.106+0000 7fb1148bf640 1 -- 172.21.15.3:0/1849892653 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fb100001c70 con 0x7fb1100b4670 2024-03-25T18:35:56.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.106+0000 7fb117b25640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:56.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.106+0000 7fb117b25640 4 set_mon_vals no callback set 2024-03-25T18:35:56.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.106+0000 7fb1148bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:56.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.106+0000 7fb1148bf640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:56.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.106+0000 7fb1148bf640 10 monclient: dump: 2024-03-25T18:35:56.108 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:56.108 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:56.108 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:56.108 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:56.108 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:56.108 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:56.108 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:56.108 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:56.108 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:56.109 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:56.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.107+0000 7fb117b25640 10 monclient: shutdown 2024-03-25T18:35:56.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.107+0000 7fb117b25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:56.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.107+0000 7fb117b25640 1 -- 172.21.15.3:0/1849892653 >> v1:172.21.15.3:6789/0 conn(0x7fb1100b4670 legacy=0x7fb1101425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:56.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.107+0000 7fb117b25640 1 -- 172.21.15.3:0/1849892653 shutdown_connections 2024-03-25T18:35:56.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.107+0000 7fb117b25640 1 -- 172.21.15.3:0/1849892653 >> 172.21.15.3:0/1849892653 conn(0x7fb1100617a0 msgr2=0x7fb110061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:56.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.107+0000 7fb117b25640 1 -- 172.21.15.3:0/1849892653 shutdown_connections 2024-03-25T18:35:56.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.107+0000 7fb117b25640 1 -- 172.21.15.3:0/1849892653 wait complete. 2024-03-25T18:35:56.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.108+0000 7fb117b25640 10 monclient: build_initial_monmap 2024-03-25T18:35:56.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.108+0000 7fb117b25640 10 monclient: monmap: 2024-03-25T18:35:56.111 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:35:56.111 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:56.111 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:35:56.112 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:35:56.112 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:56.112 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:35:56.112 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:35:56.112 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:35:56.112 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:35:56.112 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:56.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.108+0000 7fb117b25640 1 librados: starting msgr at 2024-03-25T18:35:56.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.108+0000 7fb117b25640 1 librados: starting objecter 2024-03-25T18:35:56.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.108+0000 7fb117b25640 1 Processor -- start 2024-03-25T18:35:56.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.108+0000 7fb117b25640 1 -- start start 2024-03-25T18:35:56.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.108+0000 7fb117b25640 1 librados: setting wanted keys 2024-03-25T18:35:56.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.108+0000 7fb117b25640 1 librados: calling monclient init 2024-03-25T18:35:56.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.108+0000 7fb117b25640 10 monclient: init 2024-03-25T18:35:56.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.109+0000 7fb117b25640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:56.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.109+0000 7fb117b25640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:35:56.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.109+0000 7fb117b25640 10 monclient(hunting): picked mon.noname-a con 0x7fb1100b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:35:56.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.109+0000 7fb117b25640 10 monclient(hunting): picked mon.noname-b con 0x7fb1100b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:35:56.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.109+0000 7fb117b25640 10 monclient(hunting): picked mon.noname-c con 0x7fb11013c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:35:56.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.109+0000 7fb117b25640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb110143790 con 0x7fb1100b4670 2024-03-25T18:35:56.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.109+0000 7fb117b25640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1101446c0 con 0x7fb1100b4ff0 2024-03-25T18:35:56.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.109+0000 7fb117b25640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb110144930 con 0x7fb11013c480 2024-03-25T18:35:56.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.109+0000 7fb117b25640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:56.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.109+0000 7fb117b25640 10 monclient(hunting): authenticate will time out at 8010.369789s 2024-03-25T18:35:56.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.109+0000 7fb1150c0640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb1100b4ff0 0x7fb110145820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:38062/0 (socket says 172.21.15.3:38062) 2024-03-25T18:35:56.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.110+0000 7fb1150c0640 1 -- 172.21.15.3:0/1160149224 learned_addr learned my addr 172.21.15.3:0/1160149224 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:35:56.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.110+0000 7fb1067fc640 10 client.?.objecter ms_handle_connect 0x7fb1100b4ff0 2024-03-25T18:35:56.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.110+0000 7fb1067fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:56.116 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.115+0000 7fb1067fc640 10 client.?.objecter ms_handle_connect 0x7fb11013c480 2024-03-25T18:35:56.116 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.115+0000 7fb1067fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:56.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.192+0000 7fb1067fc640 10 client.?.objecter ms_handle_connect 0x7fb1100b4670 2024-03-25T18:35:56.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.192+0000 7fb1067fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:56.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.200+0000 7fb1067fc640 1 -- 172.21.15.3:0/1160149224 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1065003303 0 0) 0x7fb110144930 con 0x7fb11013c480 2024-03-25T18:35:56.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.200+0000 7fb1067fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:56.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.200+0000 7fb1067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:56.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.200+0000 7fb1067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:56.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.200+0000 7fb1067fc640 10 monclient(hunting): my global_id is 4667 2024-03-25T18:35:56.202 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.200+0000 7fb1067fc640 1 -- 172.21.15.3:0/1160149224 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0f4004eb0 con 0x7fb11013c480 2024-03-25T18:35:56.202 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.201+0000 7fb1067fc640 1 -- 172.21.15.3:0/1160149224 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1720995908 0 0) 0x7fb110143790 con 0x7fb1100b4670 2024-03-25T18:35:56.202 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.201+0000 7fb1067fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:56.202 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.201+0000 7fb1067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:56.202 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.201+0000 7fb1067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:56.202 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.201+0000 7fb1067fc640 10 monclient(hunting): my global_id is 4725 2024-03-25T18:35:56.202 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.201+0000 7fb1067fc640 1 -- 172.21.15.3:0/1160149224 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb110144930 con 0x7fb1100b4670 2024-03-25T18:35:56.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.219+0000 7fb1067fc640 1 -- 172.21.15.3:0/1160149224 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1405312255 0 0) 0x7fb0f4004eb0 con 0x7fb11013c480 2024-03-25T18:35:56.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.219+0000 7fb1067fc640 1 -- 172.21.15.3:0/1160149224 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb110143790 con 0x7fb11013c480 2024-03-25T18:35:56.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.219+0000 7fb1067fc640 1 -- 172.21.15.3:0/1160149224 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fb108001ad0 con 0x7fb11013c480 2024-03-25T18:35:56.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.219+0000 7fb1067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:56.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.219+0000 7fb1067fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:56.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.219+0000 7fb1067fc640 10 monclient(hunting): dump: 2024-03-25T18:35:56.220 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:56.221 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:56.221 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:56.221 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:56.221 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:56.221 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:56.221 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:56.221 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:56.221 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:56.221 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:56.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.219+0000 7fb1067fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:56.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.229+0000 7fb1067fc640 1 -- 172.21.15.3:0/1160149224 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 19432652 0 0) 0x7fb110144930 con 0x7fb1100b4670 2024-03-25T18:35:56.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.229+0000 7fb1067fc640 1 -- 172.21.15.3:0/1160149224 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0f4004eb0 con 0x7fb1100b4670 2024-03-25T18:35:56.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.229+0000 7fb1067fc640 1 -- 172.21.15.3:0/1160149224 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fb10c0018b0 con 0x7fb1100b4670 2024-03-25T18:35:56.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.229+0000 7fb1067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:56.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.229+0000 7fb1067fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:56.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.229+0000 7fb1067fc640 10 monclient(hunting): dump: 2024-03-25T18:35:56.230 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:56.230 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:56.230 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:56.231 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:56.231 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:56.231 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:56.231 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:56.231 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:56.231 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:56.231 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:56.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.236+0000 7fb1067fc640 1 -- 172.21.15.3:0/1160149224 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1729038178 0 0) 0x7fb0f4004eb0 con 0x7fb1100b4670 2024-03-25T18:35:56.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.236+0000 7fb1067fc640 1 -- 172.21.15.3:0/1160149224 >> v1:172.21.15.3:6790/0 conn(0x7fb11013c480 legacy=0x7fb1101e2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:56.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.236+0000 7fb1067fc640 1 -- 172.21.15.3:0/1160149224 >> v1:172.21.15.16:6789/0 conn(0x7fb1100b4ff0 legacy=0x7fb110145820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:56.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.236+0000 7fb1067fc640 10 monclient: _finish_hunting 0 2024-03-25T18:35:56.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.236+0000 7fb1067fc640 1 monclient: found mon.a 2024-03-25T18:35:56.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.236+0000 7fb1067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:56.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.236+0000 7fb1067fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:35:56.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.236+0000 7fb1067fc640 1 -- 172.21.15.3:0/1160149224 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb110144c10 con 0x7fb1100b4670 2024-03-25T18:35:56.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.236+0000 7fb1067fc640 10 monclient: _finish_auth 0 2024-03-25T18:35:56.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.236+0000 7fb1067fc640 10 monclient: _check_auth_tickets 2024-03-25T18:35:56.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.236+0000 7fb1067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:56.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.236+0000 7fb117b25640 5 monclient: authenticate success, global_id 4725 2024-03-25T18:35:56.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.236+0000 7fb117b25640 10 monclient: _renew_subs 2024-03-25T18:35:56.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.236+0000 7fb117b25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:35:56.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.236+0000 7fb117b25640 1 -- 172.21.15.3:0/1160149224 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb11013c190 con 0x7fb1100b4670 2024-03-25T18:35:56.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.236+0000 7fb117b25640 10 client.4725.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:56.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.236+0000 7fb117b25640 10 monclient: _renew_subs 2024-03-25T18:35:56.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.236+0000 7fb117b25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:35:56.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.236+0000 7fb117b25640 1 -- 172.21.15.3:0/1160149224 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb110137340 con 0x7fb1100b4670 2024-03-25T18:35:56.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.236+0000 7fb117b25640 1 librados: init done 2024-03-25T18:35:56.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.237+0000 7fb117b25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:56.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.237+0000 7fb117b25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:56.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.237+0000 7fb117b25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:35:56.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.237+0000 7fb117b25640 1 -- 172.21.15.3:0/1160149224 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb11006dbd0 con 0x7fb1100b4670 2024-03-25T18:35:56.552 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.550+0000 7fb1067fc640 1 -- 172.21.15.3:0/1160149224 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb10c003610 con 0x7fb1100b4670 2024-03-25T18:35:56.552 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.551+0000 7fb1067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:56.552 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.551+0000 7fb1067fc640 1 -- 172.21.15.3:0/1160149224 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fb10c003a60 con 0x7fb1100b4670 2024-03-25T18:35:56.552 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.551+0000 7fb1067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:56.552 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.551+0000 7fb1067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:56.552 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.551+0000 7fb1067fc640 10 monclient: dump: 2024-03-25T18:35:56.552 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:35:56.552 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:35:56.552 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:56.553 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:35:56.553 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:56.553 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:35:56.553 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:35:56.553 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:35:56.553 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:35:56.553 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:35:56.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.551+0000 7fb107fff640 4 set_mon_vals no callback set 2024-03-25T18:35:56.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.551+0000 7fb1067fc640 1 -- 172.21.15.3:0/1160149224 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7fb10c003c80 con 0x7fb1100b4670 2024-03-25T18:35:56.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.552+0000 7fb1150c0640 10 monclient: get_auth_request con 0x7fb0f406df10 auth_method 0 2024-03-25T18:35:56.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.552+0000 7fb1067fc640 10 client.4725.objecter ms_handle_connect 0x7fb0f406df10 2024-03-25T18:35:56.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.556+0000 7fb1067fc640 1 -- 172.21.15.3:0/1160149224 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2800263165 0 0) 0x7fb10c0535f0 con 0x7fb1100b4670 2024-03-25T18:35:56.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.556+0000 7fb1067fc640 10 client.4725.objecter ms_dispatch 0x7fb110000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:35:56.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.556+0000 7fb1067fc640 3 client.4725.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:35:56.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.556+0000 7fb1067fc640 3 client.4725.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:35:56.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.557+0000 7fb1067fc640 20 client.4725.objecter dump_active .. 0 homeless 2024-03-25T18:35:56.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.557+0000 7fb1067fc640 1 -- 172.21.15.3:0/1160149224 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb10c04f100 con 0x7fb1100b4670 2024-03-25T18:35:56.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.557+0000 7fb1067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:56.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.557+0000 7fb1067fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:56.814 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.813+0000 7fb117b25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:56.814 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.813+0000 7fb117b25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:56.815 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.813+0000 7fb117b25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:35:56.815 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.813+0000 7fb117b25640 1 -- 172.21.15.3:0/1160149224 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1100688c0 con 0x7fb1100b4670 2024-03-25T18:35:56.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.846+0000 7fb1067fc640 1 -- 172.21.15.3:0/1160149224 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 716832884) 0x7fb10c051030 con 0x7fb1100b4670 2024-03-25T18:35:56.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.846+0000 7fb1067fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:56.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.846+0000 7fb1067fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:56.847 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:35:56.847 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":11,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:35:08.249841+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:35:08.249841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:56.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.848+0000 7fb117b25640 10 librados: watch_flush enter 2024-03-25T18:35:56.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.848+0000 7fb117b25640 10 librados: watch_flush exit 2024-03-25T18:35:56.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.848+0000 7fb117b25640 20 client.4725.objecter shutdown clearing up homeless session... 2024-03-25T18:35:56.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.848+0000 7fb117b25640 10 client.4725.objecter successfully canceled tick 2024-03-25T18:35:56.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.848+0000 7fb117b25640 1 -- 172.21.15.3:0/1160149224 >> v1:172.21.15.3:6800/4195140966 conn(0x7fb0f406df10 legacy=0x7fb0f40703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:56.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.848+0000 7fb117b25640 10 monclient: shutdown 2024-03-25T18:35:56.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.848+0000 7fb117b25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:56.850 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.848+0000 7fb117b25640 1 -- 172.21.15.3:0/1160149224 >> v1:172.21.15.3:6789/0 conn(0x7fb1100b4670 legacy=0x7fb110145110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:56.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.849+0000 7fb117b25640 1 -- 172.21.15.3:0/1160149224 shutdown_connections 2024-03-25T18:35:56.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.849+0000 7fb117b25640 1 -- 172.21.15.3:0/1160149224 >> 172.21.15.3:0/1160149224 conn(0x7fb1100617a0 msgr2=0x7fb1100b7030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:56.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.849+0000 7fb117b25640 1 -- 172.21.15.3:0/1160149224 shutdown_connections 2024-03-25T18:35:56.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.849+0000 7fb117b25640 1 -- 172.21.15.3:0/1160149224 wait complete. 2024-03-25T18:35:56.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:35:56.849+0000 7fb117b25640 1 librados: shutdown 2024-03-25T18:35:56.863 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:03.543 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:11:17.272 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:03.552 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:11:17.281 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:03.583 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:11:17.312 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:03.606 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:11:17.298 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:03.614 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:11:17.306 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:03.617 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:11:17.346 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:03.623 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:11:17.352 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:03.645 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:11:17.337 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:03.662 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:11:17.391 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:03.668 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:03.779 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.778+0000 7f348c027640 10 monclient: get_monmap_and_config 2024-03-25T18:36:03.780 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.778+0000 7f348c027640 10 monclient: build_initial_monmap 2024-03-25T18:36:03.780 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.778+0000 7f348c027640 10 monclient: monmap: 2024-03-25T18:36:03.780 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:36:03.780 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:36:03.780 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:36:03.779874+0000 2024-03-25T18:36:03.780 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:36:03.779874+0000 2024-03-25T18:36:03.780 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:03.780 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:36:03.780 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:36:03.780 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:36:03.780 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:36:03.780 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:36:03.781 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.779+0000 7f348c027640 1 Processor -- start 2024-03-25T18:36:03.781 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.779+0000 7f348c027640 1 -- start start 2024-03-25T18:36:03.781 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.779+0000 7f348c027640 10 monclient: init 2024-03-25T18:36:03.781 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.780+0000 7f348c027640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:03.781 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.780+0000 7f348c027640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:36:03.781 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.780+0000 7f348c027640 10 monclient(hunting): picked mon.noname-a con 0x7f34840b4e70 addr v1:172.21.15.3:6789/0 2024-03-25T18:36:03.781 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.780+0000 7f348c027640 10 monclient(hunting): picked mon.noname-c con 0x7f34840b44b0 addr v1:172.21.15.16:6789/0 2024-03-25T18:36:03.781 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.780+0000 7f348c027640 10 monclient(hunting): picked mon.noname-b con 0x7f34840b1b00 addr v1:172.21.15.3:6790/0 2024-03-25T18:36:03.781 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.780+0000 7f348c027640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34840a96f0 con 0x7f34840b4e70 2024-03-25T18:36:03.782 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.780+0000 7f348c027640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34840ab0e0 con 0x7f34840b44b0 2024-03-25T18:36:03.782 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.780+0000 7f348c027640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3484145f70 con 0x7f34840b1b00 2024-03-25T18:36:03.782 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.780+0000 7f348c027640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:03.782 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.780+0000 7f348c027640 10 monclient(hunting): authenticate will time out at 8018.040695s 2024-03-25T18:36:03.815 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.814+0000 7f3489dc3640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f34840b4e70 0x7f348413f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:46816/0 (socket says 172.21.15.3:46816) 2024-03-25T18:36:03.815 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.814+0000 7f3489dc3640 1 -- 172.21.15.3:0/4096299866 learned_addr learned my addr 172.21.15.3:0/4096299866 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:36:03.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.848+0000 7f3488dc1640 1 -- 172.21.15.3:0/4096299866 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082628837 0 0) 0x7f34840a96f0 con 0x7f34840b4e70 2024-03-25T18:36:03.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.848+0000 7f3488dc1640 10 monclient(hunting): _negotiate 2024-03-25T18:36:03.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.848+0000 7f3488dc1640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:03.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.848+0000 7f3488dc1640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:03.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.848+0000 7f3488dc1640 10 monclient(hunting): my global_id is 4728 2024-03-25T18:36:03.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.848+0000 7f3488dc1640 1 -- 172.21.15.3:0/4096299866 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3470003520 con 0x7f34840b4e70 2024-03-25T18:36:03.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.850+0000 7f3488dc1640 1 -- 172.21.15.3:0/4096299866 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 231858735 0 0) 0x7f34840ab0e0 con 0x7f34840b44b0 2024-03-25T18:36:03.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.850+0000 7f3488dc1640 10 monclient(hunting): _negotiate 2024-03-25T18:36:03.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.850+0000 7f3488dc1640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:03.852 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.850+0000 7f3488dc1640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:03.852 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.850+0000 7f3488dc1640 10 monclient(hunting): my global_id is 4666 2024-03-25T18:36:03.852 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.850+0000 7f3488dc1640 1 -- 172.21.15.3:0/4096299866 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34840a96f0 con 0x7f34840b44b0 2024-03-25T18:36:03.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.868+0000 7f3488dc1640 1 -- 172.21.15.3:0/4096299866 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2917318796 0 0) 0x7f3484145f70 con 0x7f34840b1b00 2024-03-25T18:36:03.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.868+0000 7f3488dc1640 10 monclient(hunting): _negotiate 2024-03-25T18:36:03.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.868+0000 7f3488dc1640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:03.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.868+0000 7f3488dc1640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:03.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.868+0000 7f3488dc1640 10 monclient(hunting): my global_id is 4670 2024-03-25T18:36:03.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.868+0000 7f3488dc1640 1 -- 172.21.15.3:0/4096299866 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34840ab0e0 con 0x7f34840b1b00 2024-03-25T18:36:03.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.883+0000 7f3488dc1640 1 -- 172.21.15.3:0/4096299866 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1198672315 0 0) 0x7f34840ab0e0 con 0x7f34840b1b00 2024-03-25T18:36:03.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.883+0000 7f3488dc1640 1 -- 172.21.15.3:0/4096299866 >> v1:172.21.15.16:6789/0 conn(0x7f34840b44b0 legacy=0x7f3484145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:03.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.883+0000 7f3488dc1640 1 -- 172.21.15.3:0/4096299866 >> v1:172.21.15.3:6789/0 conn(0x7f34840b4e70 legacy=0x7f348413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:03.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.883+0000 7f3488dc1640 10 monclient: _finish_hunting 0 2024-03-25T18:36:03.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.883+0000 7f3488dc1640 1 monclient: found mon.noname-b 2024-03-25T18:36:03.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.883+0000 7f3488dc1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:03.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.883+0000 7f3488dc1640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-25T18:36:03.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.883+0000 7f3488dc1640 1 -- 172.21.15.3:0/4096299866 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34840a9b80 con 0x7f34840b1b00 2024-03-25T18:36:03.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.883+0000 7f3488dc1640 10 monclient: _finish_auth 0 2024-03-25T18:36:03.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.883+0000 7f3488dc1640 10 monclient: _check_auth_tickets 2024-03-25T18:36:03.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.883+0000 7f3488dc1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:03.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.883+0000 7f3488dc1640 1 -- 172.21.15.3:0/4096299866 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f347c001930 con 0x7f34840b1b00 2024-03-25T18:36:03.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.883+0000 7f348c027640 5 monclient: authenticate success, global_id 4670 2024-03-25T18:36:03.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.883+0000 7f348c027640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:03.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.883+0000 7f3488dc1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:03.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.883+0000 7f3488dc1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:36:03.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.883+0000 7f3488dc1640 10 monclient: dump: 2024-03-25T18:36:03.885 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:36:03.885 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:36:03.885 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:03.885 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:03.885 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:03.886 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:36:03.886 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:36:03.886 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:36:03.886 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:36:03.886 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:36:03.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.883+0000 7f348c027640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:03.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.929+0000 7f3488dc1640 1 -- 172.21.15.3:0/4096299866 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f347c001a90 con 0x7f34840b1b00 2024-03-25T18:36:03.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.929+0000 7f3488dc1640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:03.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.929+0000 7f3488dc1640 1 -- 172.21.15.3:0/4096299866 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f347c001ee0 con 0x7f34840b1b00 2024-03-25T18:36:03.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.929+0000 7f348c027640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:03.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.929+0000 7f348c027640 4 set_mon_vals no callback set 2024-03-25T18:36:03.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.929+0000 7f3488dc1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:03.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.929+0000 7f3488dc1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:36:03.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.929+0000 7f3488dc1640 10 monclient: dump: 2024-03-25T18:36:03.930 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:36:03.930 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:36:03.930 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:03.930 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:03.931 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:03.931 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:36:03.931 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:36:03.931 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:36:03.931 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:36:03.931 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:36:03.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.929+0000 7f348c027640 10 monclient: shutdown 2024-03-25T18:36:03.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.929+0000 7f348c027640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:03.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.929+0000 7f348c027640 1 -- 172.21.15.3:0/4096299866 >> v1:172.21.15.3:6790/0 conn(0x7f34840b1b00 legacy=0x7f348414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:03.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.929+0000 7f348c027640 1 -- 172.21.15.3:0/4096299866 shutdown_connections 2024-03-25T18:36:03.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.929+0000 7f348c027640 1 -- 172.21.15.3:0/4096299866 >> 172.21.15.3:0/4096299866 conn(0x7f34840617a0 msgr2=0x7f3484061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:03.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.930+0000 7f348c027640 1 -- 172.21.15.3:0/4096299866 shutdown_connections 2024-03-25T18:36:03.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.930+0000 7f348c027640 1 -- 172.21.15.3:0/4096299866 wait complete. 2024-03-25T18:36:03.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.930+0000 7f348c027640 10 monclient: build_initial_monmap 2024-03-25T18:36:03.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.930+0000 7f348c027640 10 monclient: monmap: 2024-03-25T18:36:03.933 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:36:03.934 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:03.934 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:36:03.934 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:36:03.934 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:03.934 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:36:03.934 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:36:03.934 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:36:03.934 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:36:03.934 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:36:03.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.931+0000 7f348c027640 1 librados: starting msgr at 2024-03-25T18:36:03.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.931+0000 7f348c027640 1 librados: starting objecter 2024-03-25T18:36:03.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.931+0000 7f348c027640 1 Processor -- start 2024-03-25T18:36:03.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.931+0000 7f348c027640 1 -- start start 2024-03-25T18:36:03.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.931+0000 7f348c027640 1 librados: setting wanted keys 2024-03-25T18:36:03.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.931+0000 7f348c027640 1 librados: calling monclient init 2024-03-25T18:36:03.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.931+0000 7f348c027640 10 monclient: init 2024-03-25T18:36:03.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.931+0000 7f348c027640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:03.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.931+0000 7f348c027640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:36:03.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.932+0000 7f348c027640 10 monclient(hunting): picked mon.noname-b con 0x7f34840b1b00 addr v1:172.21.15.16:6789/0 2024-03-25T18:36:03.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.932+0000 7f348c027640 10 monclient(hunting): picked mon.noname-c con 0x7f34840b44b0 addr v1:172.21.15.3:6790/0 2024-03-25T18:36:03.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.932+0000 7f348c027640 10 monclient(hunting): picked mon.noname-a con 0x7f34840b4e70 addr v1:172.21.15.3:6789/0 2024-03-25T18:36:03.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.932+0000 7f348c027640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f348412e9b0 con 0x7f34840b4e70 2024-03-25T18:36:03.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.932+0000 7f348c027640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f348412fa60 con 0x7f34840b1b00 2024-03-25T18:36:03.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.932+0000 7f348c027640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34841311a0 con 0x7f34840b44b0 2024-03-25T18:36:03.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.932+0000 7f348c027640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:03.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.932+0000 7f348c027640 10 monclient(hunting): authenticate will time out at 8018.192585s 2024-03-25T18:36:03.941 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.940+0000 7f3489dc3640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f34840b1b00 0x7f348412f350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:58962/0 (socket says 172.21.15.3:58962) 2024-03-25T18:36:03.941 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.940+0000 7f3489dc3640 1 -- 172.21.15.3:0/1170788309 learned_addr learned my addr 172.21.15.3:0/1170788309 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:36:03.942 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.941+0000 7f347a7fc640 10 client.?.objecter ms_handle_connect 0x7f34840b1b00 2024-03-25T18:36:03.942 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.941+0000 7f347a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:03.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.953+0000 7f347a7fc640 10 client.?.objecter ms_handle_connect 0x7f34840b44b0 2024-03-25T18:36:03.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.953+0000 7f347a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:03.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.953+0000 7f347a7fc640 10 client.?.objecter ms_handle_connect 0x7f34840b4e70 2024-03-25T18:36:03.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.953+0000 7f347a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:03.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.962+0000 7f347a7fc640 1 -- 172.21.15.3:0/1170788309 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886612688 0 0) 0x7f348412e9b0 con 0x7f34840b4e70 2024-03-25T18:36:03.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.962+0000 7f347a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:03.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.962+0000 7f347a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:03.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.962+0000 7f347a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:03.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.962+0000 7f347a7fc640 10 monclient(hunting): my global_id is 4731 2024-03-25T18:36:03.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.963+0000 7f347a7fc640 1 -- 172.21.15.3:0/1170788309 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3468004eb0 con 0x7f34840b4e70 2024-03-25T18:36:03.972 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.971+0000 7f347a7fc640 1 -- 172.21.15.3:0/1170788309 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2172002874 0 0) 0x7f3470003520 con 0x7f34840b1b00 2024-03-25T18:36:03.972 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.971+0000 7f347a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:03.972 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.971+0000 7f347a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:03.972 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.971+0000 7f347a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:03.972 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.971+0000 7f347a7fc640 10 monclient(hunting): my global_id is 4669 2024-03-25T18:36:03.972 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.971+0000 7f347a7fc640 1 -- 172.21.15.3:0/1170788309 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f348412e9b0 con 0x7f34840b1b00 2024-03-25T18:36:03.978 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.977+0000 7f347a7fc640 1 -- 172.21.15.3:0/1170788309 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 482737603 0 0) 0x7f34841311a0 con 0x7f34840b44b0 2024-03-25T18:36:03.978 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.977+0000 7f347a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:03.978 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.977+0000 7f347a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:03.978 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.977+0000 7f347a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:03.979 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.977+0000 7f347a7fc640 10 monclient(hunting): my global_id is 4673 2024-03-25T18:36:03.979 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.977+0000 7f347a7fc640 1 -- 172.21.15.3:0/1170788309 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3470003520 con 0x7f34840b44b0 2024-03-25T18:36:03.986 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.985+0000 7f347a7fc640 1 -- 172.21.15.3:0/1170788309 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1956420334 0 0) 0x7f348412e9b0 con 0x7f34840b1b00 2024-03-25T18:36:03.986 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.985+0000 7f347a7fc640 1 -- 172.21.15.3:0/1170788309 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34841311a0 con 0x7f34840b1b00 2024-03-25T18:36:03.986 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.985+0000 7f347a7fc640 1 -- 172.21.15.3:0/1170788309 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f34800016d0 con 0x7f34840b1b00 2024-03-25T18:36:03.986 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.985+0000 7f347a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:03.986 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.985+0000 7f347a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:36:03.986 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.985+0000 7f347a7fc640 10 monclient(hunting): dump: 2024-03-25T18:36:03.987 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:36:03.987 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:36:03.987 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:03.987 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:03.987 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:03.987 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:36:03.987 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:36:03.987 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:36:03.987 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:36:03.987 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:36:03.987 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.985+0000 7f347a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:03.997 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.996+0000 7f347a7fc640 1 -- 172.21.15.3:0/1170788309 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 462689785 0 0) 0x7f3468004eb0 con 0x7f34840b4e70 2024-03-25T18:36:03.997 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.996+0000 7f347a7fc640 1 -- 172.21.15.3:0/1170788309 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f348412e9b0 con 0x7f34840b4e70 2024-03-25T18:36:03.997 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.996+0000 7f347a7fc640 1 -- 172.21.15.3:0/1170788309 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f347c001da0 con 0x7f34840b4e70 2024-03-25T18:36:03.997 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.996+0000 7f347a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:03.997 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.996+0000 7f347a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:03.997 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:03.996+0000 7f347a7fc640 10 monclient(hunting): dump: 2024-03-25T18:36:03.997 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:36:03.997 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:36:03.997 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:03.998 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:03.998 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:03.998 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:36:03.998 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:36:03.998 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:36:03.998 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:36:03.998 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:36:04.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.005+0000 7f347a7fc640 1 -- 172.21.15.3:0/1170788309 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 48825113 0 0) 0x7f34841311a0 con 0x7f34840b1b00 2024-03-25T18:36:04.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.005+0000 7f347a7fc640 1 -- 172.21.15.3:0/1170788309 >> v1:172.21.15.3:6790/0 conn(0x7f34840b44b0 legacy=0x7f348412d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:04.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.005+0000 7f347a7fc640 1 -- 172.21.15.3:0/1170788309 >> v1:172.21.15.3:6789/0 conn(0x7f34840b4e70 legacy=0x7f348413b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:04.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.005+0000 7f347a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:36:04.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.005+0000 7f347a7fc640 1 monclient: found mon.b 2024-03-25T18:36:04.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.005+0000 7f347a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:04.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.005+0000 7f347a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:36:04.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.005+0000 7f347a7fc640 1 -- 172.21.15.3:0/1170788309 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f348412e640 con 0x7f34840b1b00 2024-03-25T18:36:04.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.005+0000 7f347a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:36:04.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.005+0000 7f347a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:36:04.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.005+0000 7f347a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:04.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.005+0000 7f348c027640 5 monclient: authenticate success, global_id 4669 2024-03-25T18:36:04.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.005+0000 7f348c027640 10 monclient: _renew_subs 2024-03-25T18:36:04.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.005+0000 7f348c027640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:36:04.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.005+0000 7f348c027640 1 -- 172.21.15.3:0/1170788309 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f348413c810 con 0x7f34840b1b00 2024-03-25T18:36:04.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.005+0000 7f348c027640 10 client.4669.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:04.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.005+0000 7f348c027640 10 monclient: _renew_subs 2024-03-25T18:36:04.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.005+0000 7f348c027640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:36:04.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.005+0000 7f348c027640 1 -- 172.21.15.3:0/1170788309 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f348412d9c0 con 0x7f34840b1b00 2024-03-25T18:36:04.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.005+0000 7f348c027640 1 librados: init done 2024-03-25T18:36:04.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.006+0000 7f348c027640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:04.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.006+0000 7f348c027640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:04.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.006+0000 7f348c027640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:36:04.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.006+0000 7f348c027640 1 -- 172.21.15.3:0/1170788309 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f348404f730 con 0x7f34840b1b00 2024-03-25T18:36:04.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.300+0000 7f347a7fc640 1 -- 172.21.15.3:0/1170788309 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3480001ba0 con 0x7f34840b1b00 2024-03-25T18:36:04.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.300+0000 7f347a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:04.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.300+0000 7f347a7fc640 1 -- 172.21.15.3:0/1170788309 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f3480004980 con 0x7f34840b1b00 2024-03-25T18:36:04.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.300+0000 7f347a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:04.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.300+0000 7f347a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:04.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.300+0000 7f347a7fc640 10 monclient: dump: 2024-03-25T18:36:04.302 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:36:04.302 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:36:04.302 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:04.302 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:04.302 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:04.302 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:36:04.302 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:36:04.302 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:36:04.302 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:36:04.302 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:36:04.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.300+0000 7f347bfff640 4 set_mon_vals no callback set 2024-03-25T18:36:04.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.301+0000 7f347a7fc640 1 -- 172.21.15.3:0/1170788309 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f348001a950 con 0x7f34840b1b00 2024-03-25T18:36:04.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.301+0000 7f34895c2640 10 monclient: get_auth_request con 0x7f346806df70 auth_method 0 2024-03-25T18:36:04.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.302+0000 7f347a7fc640 10 client.4669.objecter ms_handle_connect 0x7f346806df70 2024-03-25T18:36:04.306 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.305+0000 7f347a7fc640 1 -- 172.21.15.3:0/1170788309 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2800263165 0 0) 0x7f3480001d10 con 0x7f34840b1b00 2024-03-25T18:36:04.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.305+0000 7f347a7fc640 10 client.4669.objecter ms_dispatch 0x7f3484000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:36:04.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.305+0000 7f347a7fc640 3 client.4669.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:36:04.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.305+0000 7f347a7fc640 3 client.4669.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:36:04.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.305+0000 7f347a7fc640 20 client.4669.objecter dump_active .. 0 homeless 2024-03-25T18:36:04.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.306+0000 7f347a7fc640 1 -- 172.21.15.3:0/1170788309 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f348004fef0 con 0x7f34840b1b00 2024-03-25T18:36:04.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.306+0000 7f347a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:04.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.306+0000 7f347a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:04.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.556+0000 7f348c027640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:04.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.556+0000 7f348c027640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:04.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.556+0000 7f348c027640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:36:04.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.556+0000 7f348c027640 1 -- 172.21.15.3:0/1170788309 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34840aa090 con 0x7f34840b1b00 2024-03-25T18:36:04.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.632+0000 7f347a7fc640 1 -- 172.21.15.3:0/1170788309 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 716832884) 0x7f3480051ff0 con 0x7f34840b1b00 2024-03-25T18:36:04.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.632+0000 7f347a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:04.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.632+0000 7f347a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:04.633 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:36:04.633 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":11,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:35:08.249841+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:35:08.249841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:04.636 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.634+0000 7f348c027640 10 librados: watch_flush enter 2024-03-25T18:36:04.636 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.634+0000 7f348c027640 10 librados: watch_flush exit 2024-03-25T18:36:04.636 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.634+0000 7f348c027640 20 client.4669.objecter shutdown clearing up homeless session... 2024-03-25T18:36:04.636 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.634+0000 7f348c027640 10 client.4669.objecter successfully canceled tick 2024-03-25T18:36:04.636 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.635+0000 7f348c027640 1 -- 172.21.15.3:0/1170788309 >> v1:172.21.15.3:6800/4195140966 conn(0x7f346806df70 legacy=0x7f3468070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:04.636 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.635+0000 7f348c027640 10 monclient: shutdown 2024-03-25T18:36:04.636 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.635+0000 7f348c027640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:04.636 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.635+0000 7f348c027640 1 -- 172.21.15.3:0/1170788309 >> v1:172.21.15.16:6789/0 conn(0x7f34840b1b00 legacy=0x7f348412f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:04.637 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.635+0000 7f348c027640 1 -- 172.21.15.3:0/1170788309 shutdown_connections 2024-03-25T18:36:04.637 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.635+0000 7f348c027640 1 -- 172.21.15.3:0/1170788309 >> 172.21.15.3:0/1170788309 conn(0x7f34840617a0 msgr2=0x7f34840b66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:04.637 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.635+0000 7f348c027640 1 -- 172.21.15.3:0/1170788309 shutdown_connections 2024-03-25T18:36:04.637 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.635+0000 7f348c027640 1 -- 172.21.15.3:0/1170788309 wait complete. 2024-03-25T18:36:04.637 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:04.635+0000 7f348c027640 1 librados: shutdown 2024-03-25T18:36:04.646 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:05.915 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:11:19.644 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:05.921 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:11:19.650 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:05.928 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:11:19.658 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:05.929 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:11:19.621 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:05.932 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:11:19.661 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:05.934 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:11:19.626 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:05.943 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:11:19.672 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:05.949 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:11:19.679 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:06.016 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:11:19.708 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:11.452 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:11.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.566+0000 7f216d713640 10 monclient: get_monmap_and_config 2024-03-25T18:36:11.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.566+0000 7f216d713640 10 monclient: build_initial_monmap 2024-03-25T18:36:11.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.566+0000 7f216d713640 10 monclient: monmap: 2024-03-25T18:36:11.567 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:36:11.567 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:36:11.567 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:36:11.567530+0000 2024-03-25T18:36:11.567 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:36:11.567530+0000 2024-03-25T18:36:11.568 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:11.568 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:36:11.568 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:36:11.568 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:36:11.568 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:36:11.568 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:36:11.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.567+0000 7f216d713640 1 Processor -- start 2024-03-25T18:36:11.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.567+0000 7f216d713640 1 -- start start 2024-03-25T18:36:11.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.567+0000 7f216d713640 10 monclient: init 2024-03-25T18:36:11.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.567+0000 7f216d713640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:11.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.567+0000 7f216d713640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:36:11.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.567+0000 7f216d713640 10 monclient(hunting): picked mon.noname-c con 0x7f21680b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:36:11.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.567+0000 7f216d713640 10 monclient(hunting): picked mon.noname-a con 0x7f21680b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:36:11.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.567+0000 7f216d713640 10 monclient(hunting): picked mon.noname-b con 0x7f216813c480 addr v1:172.21.15.3:6790/0 2024-03-25T18:36:11.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.567+0000 7f216d713640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21680ab0e0 con 0x7f21680b4670 2024-03-25T18:36:11.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.567+0000 7f216d713640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21680ab3c0 con 0x7f21680b4ff0 2024-03-25T18:36:11.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.567+0000 7f216d713640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21680a9f30 con 0x7f216813c480 2024-03-25T18:36:11.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.567+0000 7f216d713640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:11.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.567+0000 7f216d713640 10 monclient(hunting): authenticate will time out at 8025.828310s 2024-03-25T18:36:11.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.575+0000 7f2166ffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f21680b4ff0 0x7f216813bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:47944/0 (socket says 172.21.15.3:47944) 2024-03-25T18:36:11.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.575+0000 7f2166ffd640 1 -- 172.21.15.3:0/1457246881 learned_addr learned my addr 172.21.15.3:0/1457246881 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:36:11.588 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.587+0000 7f21667fc640 1 -- 172.21.15.3:0/1457246881 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4074819971 0 0) 0x7f21680ab0e0 con 0x7f21680b4670 2024-03-25T18:36:11.588 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.587+0000 7f21667fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:11.589 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.587+0000 7f21667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:11.589 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.587+0000 7f21667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:11.589 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.587+0000 7f21667fc640 10 monclient(hunting): my global_id is 4734 2024-03-25T18:36:11.589 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.587+0000 7f21667fc640 1 -- 172.21.15.3:0/1457246881 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2154003520 con 0x7f21680b4670 2024-03-25T18:36:11.589 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.587+0000 7f21667fc640 1 -- 172.21.15.3:0/1457246881 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 376811304 0 0) 0x7f21680ab3c0 con 0x7f21680b4ff0 2024-03-25T18:36:11.589 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.587+0000 7f21667fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:11.589 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.587+0000 7f21667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:11.589 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.587+0000 7f21667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:11.589 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.587+0000 7f21667fc640 10 monclient(hunting): my global_id is 4672 2024-03-25T18:36:11.589 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.587+0000 7f21667fc640 1 -- 172.21.15.3:0/1457246881 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21680ab0e0 con 0x7f21680b4ff0 2024-03-25T18:36:11.591 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.590+0000 7f21667fc640 1 -- 172.21.15.3:0/1457246881 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2968640778 0 0) 0x7f21680a9f30 con 0x7f216813c480 2024-03-25T18:36:11.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.590+0000 7f21667fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:11.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.590+0000 7f21667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:11.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.590+0000 7f21667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:11.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.590+0000 7f21667fc640 10 monclient(hunting): my global_id is 4676 2024-03-25T18:36:11.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.591+0000 7f21667fc640 1 -- 172.21.15.3:0/1457246881 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21680ab3c0 con 0x7f216813c480 2024-03-25T18:36:11.597 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.596+0000 7f21667fc640 1 -- 172.21.15.3:0/1457246881 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3625981139 0 0) 0x7f2154003520 con 0x7f21680b4670 2024-03-25T18:36:11.597 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.596+0000 7f21667fc640 1 -- 172.21.15.3:0/1457246881 >> v1:172.21.15.3:6790/0 conn(0x7f216813c480 legacy=0x7f216814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:11.597 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.596+0000 7f21667fc640 1 -- 172.21.15.3:0/1457246881 >> v1:172.21.15.16:6789/0 conn(0x7f21680b4ff0 legacy=0x7f216813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:11.597 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.596+0000 7f21667fc640 10 monclient: _finish_hunting 0 2024-03-25T18:36:11.597 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.596+0000 7f21667fc640 1 monclient: found mon.noname-a 2024-03-25T18:36:11.597 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.596+0000 7f21667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:11.598 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.596+0000 7f21667fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:36:11.598 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.596+0000 7f21667fc640 1 -- 172.21.15.3:0/1457246881 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21680a9b70 con 0x7f21680b4670 2024-03-25T18:36:11.598 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.596+0000 7f21667fc640 10 monclient: _finish_auth 0 2024-03-25T18:36:11.598 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.596+0000 7f21667fc640 10 monclient: _check_auth_tickets 2024-03-25T18:36:11.598 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.596+0000 7f21667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:11.598 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.596+0000 7f21667fc640 1 -- 172.21.15.3:0/1457246881 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f2150001690 con 0x7f21680b4670 2024-03-25T18:36:11.598 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.596+0000 7f21667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:11.598 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.596+0000 7f21667fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:36:11.598 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.596+0000 7f21667fc640 10 monclient: dump: 2024-03-25T18:36:11.598 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:36:11.598 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:36:11.598 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:11.598 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:11.598 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:11.599 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:36:11.599 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:36:11.599 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:36:11.599 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:36:11.599 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:36:11.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.596+0000 7f216d713640 5 monclient: authenticate success, global_id 4734 2024-03-25T18:36:11.599 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.596+0000 7f216d713640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:11.617 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.616+0000 7f21667fc640 1 -- 172.21.15.3:0/1457246881 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21500017f0 con 0x7f21680b4670 2024-03-25T18:36:11.617 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.616+0000 7f21667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:11.617 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.616+0000 7f21667fc640 1 -- 172.21.15.3:0/1457246881 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f2150001c00 con 0x7f21680b4670 2024-03-25T18:36:11.617 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.616+0000 7f21667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:11.617 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.616+0000 7f21667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:11.617 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.616+0000 7f21667fc640 10 monclient: dump: 2024-03-25T18:36:11.617 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:36:11.617 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:36:11.617 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:11.618 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:11.618 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:11.618 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:36:11.618 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:36:11.618 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:36:11.618 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:36:11.618 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:36:11.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.616+0000 7f216d713640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:11.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.616+0000 7f216d713640 4 set_mon_vals no callback set 2024-03-25T18:36:11.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.617+0000 7f216d713640 10 monclient: shutdown 2024-03-25T18:36:11.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.617+0000 7f216d713640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:11.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.617+0000 7f216d713640 1 -- 172.21.15.3:0/1457246881 >> v1:172.21.15.3:6789/0 conn(0x7f21680b4670 legacy=0x7f21681425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:11.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.617+0000 7f216d713640 1 -- 172.21.15.3:0/1457246881 shutdown_connections 2024-03-25T18:36:11.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.617+0000 7f216d713640 1 -- 172.21.15.3:0/1457246881 >> 172.21.15.3:0/1457246881 conn(0x7f21680617a0 msgr2=0x7f2168061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:11.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.617+0000 7f216d713640 1 -- 172.21.15.3:0/1457246881 shutdown_connections 2024-03-25T18:36:11.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.617+0000 7f216d713640 1 -- 172.21.15.3:0/1457246881 wait complete. 2024-03-25T18:36:11.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.618+0000 7f216d713640 10 monclient: build_initial_monmap 2024-03-25T18:36:11.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.618+0000 7f216d713640 10 monclient: monmap: 2024-03-25T18:36:11.620 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:36:11.620 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:11.620 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:36:11.620 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:36:11.620 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:11.620 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:36:11.620 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:36:11.620 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:36:11.620 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:36:11.621 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:36:11.621 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.618+0000 7f216d713640 1 librados: starting msgr at 2024-03-25T18:36:11.621 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.618+0000 7f216d713640 1 librados: starting objecter 2024-03-25T18:36:11.621 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.618+0000 7f216d713640 1 Processor -- start 2024-03-25T18:36:11.621 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.618+0000 7f216d713640 1 -- start start 2024-03-25T18:36:11.621 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.618+0000 7f216d713640 1 librados: setting wanted keys 2024-03-25T18:36:11.621 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.618+0000 7f216d713640 1 librados: calling monclient init 2024-03-25T18:36:11.621 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.618+0000 7f216d713640 10 monclient: init 2024-03-25T18:36:11.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.619+0000 7f216d713640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:11.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.619+0000 7f216d713640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:36:11.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.619+0000 7f216d713640 10 monclient(hunting): picked mon.noname-a con 0x7f21680b4670 addr v1:172.21.15.3:6789/0 2024-03-25T18:36:11.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.619+0000 7f216d713640 10 monclient(hunting): picked mon.noname-c con 0x7f21680b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:36:11.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.619+0000 7f216d713640 10 monclient(hunting): picked mon.noname-b con 0x7f216813c480 addr v1:172.21.15.16:6789/0 2024-03-25T18:36:11.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.619+0000 7f216d713640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21680aedf0 con 0x7f21680b4670 2024-03-25T18:36:11.623 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.619+0000 7f216d713640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21680b0220 con 0x7f216813c480 2024-03-25T18:36:11.623 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.620+0000 7f216d713640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21680b0500 con 0x7f21680b4ff0 2024-03-25T18:36:11.623 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.620+0000 7f216d713640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:11.623 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.620+0000 7f216d713640 10 monclient(hunting): authenticate will time out at 8025.880434s 2024-03-25T18:36:11.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.626+0000 7f215ffff640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f21680b4ff0 0x7f21680b0c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:45784/0 (socket says 172.21.15.3:45784) 2024-03-25T18:36:11.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.626+0000 7f215ffff640 1 -- 172.21.15.3:0/1078209263 learned_addr learned my addr 172.21.15.3:0/1078209263 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:36:11.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.630+0000 7f215f7fe640 10 client.?.objecter ms_handle_connect 0x7f21680b4670 2024-03-25T18:36:11.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.630+0000 7f215f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:11.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.633+0000 7f215f7fe640 10 client.?.objecter ms_handle_connect 0x7f21680b4ff0 2024-03-25T18:36:11.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.633+0000 7f215f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:11.639 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.638+0000 7f215f7fe640 10 client.?.objecter ms_handle_connect 0x7f216813c480 2024-03-25T18:36:11.639 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.638+0000 7f215f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:11.640 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.638+0000 7f215f7fe640 1 -- 172.21.15.3:0/1078209263 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3572051487 0 0) 0x7f21680b0500 con 0x7f21680b4ff0 2024-03-25T18:36:11.640 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.638+0000 7f215f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:36:11.640 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.638+0000 7f215f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:11.640 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.638+0000 7f215f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:11.640 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.638+0000 7f215f7fe640 10 monclient(hunting): my global_id is 4679 2024-03-25T18:36:11.640 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.639+0000 7f215f7fe640 1 -- 172.21.15.3:0/1078209263 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2148004ee0 con 0x7f21680b4ff0 2024-03-25T18:36:11.648 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.647+0000 7f215f7fe640 1 -- 172.21.15.3:0/1078209263 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2181857135 0 0) 0x7f2148004ee0 con 0x7f21680b4ff0 2024-03-25T18:36:11.648 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.647+0000 7f215f7fe640 1 -- 172.21.15.3:0/1078209263 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21680b0500 con 0x7f21680b4ff0 2024-03-25T18:36:11.648 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.647+0000 7f215f7fe640 1 -- 172.21.15.3:0/1078209263 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f2150001bc0 con 0x7f21680b4ff0 2024-03-25T18:36:11.648 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.647+0000 7f215f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:11.648 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.647+0000 7f215f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:36:11.648 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.647+0000 7f215f7fe640 10 monclient(hunting): dump: 2024-03-25T18:36:11.649 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:36:11.649 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:36:11.649 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:11.649 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:11.649 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:11.649 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:36:11.649 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:36:11.649 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:36:11.649 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:36:11.649 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:36:11.649 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.647+0000 7f215f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:11.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.650+0000 7f215f7fe640 1 -- 172.21.15.3:0/1078209263 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2839377151 0 0) 0x7f21680b0220 con 0x7f216813c480 2024-03-25T18:36:11.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.650+0000 7f215f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:36:11.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.650+0000 7f215f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:11.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.650+0000 7f215f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:11.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.650+0000 7f215f7fe640 10 monclient(hunting): my global_id is 4675 2024-03-25T18:36:11.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.650+0000 7f215f7fe640 1 -- 172.21.15.3:0/1078209263 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2148004ee0 con 0x7f216813c480 2024-03-25T18:36:11.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.658+0000 7f215f7fe640 1 -- 172.21.15.3:0/1078209263 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2428275425 0 0) 0x7f21680aedf0 con 0x7f21680b4670 2024-03-25T18:36:11.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.658+0000 7f215f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:36:11.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.658+0000 7f215f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:11.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.658+0000 7f215f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:11.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.658+0000 7f215f7fe640 10 monclient(hunting): my global_id is 4737 2024-03-25T18:36:11.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.658+0000 7f215f7fe640 1 -- 172.21.15.3:0/1078209263 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21680b0220 con 0x7f21680b4670 2024-03-25T18:36:11.663 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.662+0000 7f215f7fe640 1 -- 172.21.15.3:0/1078209263 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1892347717 0 0) 0x7f21680b0500 con 0x7f21680b4ff0 2024-03-25T18:36:11.663 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.662+0000 7f215f7fe640 1 -- 172.21.15.3:0/1078209263 >> v1:172.21.15.16:6789/0 conn(0x7f216813c480 legacy=0x7f21681e2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:11.663 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.662+0000 7f215f7fe640 1 -- 172.21.15.3:0/1078209263 >> v1:172.21.15.3:6789/0 conn(0x7f21680b4670 legacy=0x7f21680ae380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:11.663 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.662+0000 7f215f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:36:11.663 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.662+0000 7f215f7fe640 1 monclient: found mon.c 2024-03-25T18:36:11.663 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.662+0000 7f215f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:11.664 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.662+0000 7f215f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:36:11.664 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.662+0000 7f215f7fe640 1 -- 172.21.15.3:0/1078209263 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21680b07e0 con 0x7f21680b4ff0 2024-03-25T18:36:11.664 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.662+0000 7f215f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:36:11.664 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.662+0000 7f215f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:36:11.664 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.662+0000 7f215f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:11.664 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.662+0000 7f216d713640 5 monclient: authenticate success, global_id 4679 2024-03-25T18:36:11.664 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.662+0000 7f216d713640 10 monclient: _renew_subs 2024-03-25T18:36:11.664 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.662+0000 7f216d713640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:36:11.664 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.662+0000 7f216d713640 1 -- 172.21.15.3:0/1078209263 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21680b1310 con 0x7f21680b4ff0 2024-03-25T18:36:11.664 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.662+0000 7f216d713640 10 client.4679.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:11.664 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.662+0000 7f216d713640 10 monclient: _renew_subs 2024-03-25T18:36:11.664 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.662+0000 7f216d713640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:36:11.664 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.662+0000 7f216d713640 1 -- 172.21.15.3:0/1078209263 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f21680af460 con 0x7f21680b4ff0 2024-03-25T18:36:11.664 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.662+0000 7f216d713640 1 librados: init done 2024-03-25T18:36:11.665 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.663+0000 7f216d713640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:11.665 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.663+0000 7f216d713640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:11.665 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.663+0000 7f216d713640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:36:11.665 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.663+0000 7f216d713640 1 -- 172.21.15.3:0/1078209263 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f216806dbd0 con 0x7f21680b4ff0 2024-03-25T18:36:11.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.969+0000 7f215f7fe640 1 -- 172.21.15.3:0/1078209263 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21500011d0 con 0x7f21680b4ff0 2024-03-25T18:36:11.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.969+0000 7f215f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:11.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.969+0000 7f215f7fe640 1 -- 172.21.15.3:0/1078209263 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f2150003b00 con 0x7f21680b4ff0 2024-03-25T18:36:11.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.969+0000 7f215f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:11.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.969+0000 7f215f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:36:11.971 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.969+0000 7f215f7fe640 10 monclient: dump: 2024-03-25T18:36:11.971 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:36:11.971 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:36:11.971 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:11.971 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:11.971 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:11.971 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:36:11.971 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:36:11.971 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:36:11.971 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:36:11.971 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:36:11.971 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.969+0000 7f2165ffb640 4 set_mon_vals no callback set 2024-03-25T18:36:11.972 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.970+0000 7f215f7fe640 1 -- 172.21.15.3:0/1078209263 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f2150003d20 con 0x7f21680b4ff0 2024-03-25T18:36:11.972 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.971+0000 7f2166ffd640 10 monclient: get_auth_request con 0x7f214806ded0 auth_method 0 2024-03-25T18:36:11.972 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.971+0000 7f215f7fe640 10 client.4679.objecter ms_handle_connect 0x7f214806ded0 2024-03-25T18:36:11.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.976+0000 7f215f7fe640 1 -- 172.21.15.3:0/1078209263 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2800263165 0 0) 0x7f2150054930 con 0x7f21680b4ff0 2024-03-25T18:36:11.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.976+0000 7f215f7fe640 10 client.4679.objecter ms_dispatch 0x7f2168000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:36:11.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.976+0000 7f215f7fe640 3 client.4679.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:36:11.978 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.976+0000 7f215f7fe640 3 client.4679.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:36:11.978 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.976+0000 7f215f7fe640 20 client.4679.objecter dump_active .. 0 homeless 2024-03-25T18:36:11.978 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.976+0000 7f215f7fe640 1 -- 172.21.15.3:0/1078209263 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2150050270 con 0x7f21680b4ff0 2024-03-25T18:36:11.978 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.976+0000 7f215f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:11.978 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:11.976+0000 7f215f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:12.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:12.217+0000 7f216d713640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:12.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:12.217+0000 7f216d713640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:12.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:12.217+0000 7f216d713640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-25T18:36:12.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:12.217+0000 7f216d713640 1 -- 172.21.15.3:0/1078209263 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21680688c0 con 0x7f21680b4ff0 2024-03-25T18:36:12.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:12.251+0000 7f215f7fe640 1 -- 172.21.15.3:0/1078209263 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 716832884) 0x7f2150052370 con 0x7f21680b4ff0 2024-03-25T18:36:12.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:12.251+0000 7f215f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:12.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:12.251+0000 7f215f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:12.253 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:36:12.253 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":11,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:35:08.249841+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:35:08.249841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:12.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:12.253+0000 7f216d713640 10 librados: watch_flush enter 2024-03-25T18:36:12.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:12.253+0000 7f216d713640 10 librados: watch_flush exit 2024-03-25T18:36:12.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:12.253+0000 7f216d713640 20 client.4679.objecter shutdown clearing up homeless session... 2024-03-25T18:36:12.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:12.253+0000 7f216d713640 10 client.4679.objecter successfully canceled tick 2024-03-25T18:36:12.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:12.253+0000 7f216d713640 1 -- 172.21.15.3:0/1078209263 >> v1:172.21.15.3:6800/4195140966 conn(0x7f214806ded0 legacy=0x7f2148070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:12.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:12.253+0000 7f216d713640 10 monclient: shutdown 2024-03-25T18:36:12.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:12.253+0000 7f216d713640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:12.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:12.253+0000 7f216d713640 1 -- 172.21.15.3:0/1078209263 >> v1:172.21.15.3:6790/0 conn(0x7f21680b4ff0 legacy=0x7f21680b0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:12.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:12.253+0000 7f216d713640 1 -- 172.21.15.3:0/1078209263 shutdown_connections 2024-03-25T18:36:12.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:12.253+0000 7f216d713640 1 -- 172.21.15.3:0/1078209263 >> 172.21.15.3:0/1078209263 conn(0x7f21680617a0 msgr2=0x7f21680b2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:12.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:12.253+0000 7f216d713640 1 -- 172.21.15.3:0/1078209263 shutdown_connections 2024-03-25T18:36:12.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:12.253+0000 7f216d713640 1 -- 172.21.15.3:0/1078209263 wait complete. 2024-03-25T18:36:12.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:12.253+0000 7f216d713640 1 librados: shutdown 2024-03-25T18:36:12.266 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:19.074 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:19.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.191+0000 7f81da7d6640 10 monclient: get_monmap_and_config 2024-03-25T18:36:19.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.191+0000 7f81da7d6640 10 monclient: build_initial_monmap 2024-03-25T18:36:19.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.191+0000 7f81da7d6640 10 monclient: monmap: 2024-03-25T18:36:19.192 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:36:19.192 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:36:19.192 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:36:19.192515+0000 2024-03-25T18:36:19.192 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:36:19.192515+0000 2024-03-25T18:36:19.192 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:19.192 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:36:19.193 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:36:19.193 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:36:19.193 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:36:19.193 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:36:19.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.192+0000 7f81da7d6640 1 Processor -- start 2024-03-25T18:36:19.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.192+0000 7f81da7d6640 1 -- start start 2024-03-25T18:36:19.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.192+0000 7f81da7d6640 10 monclient: init 2024-03-25T18:36:19.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.193+0000 7f81da7d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:19.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.193+0000 7f81da7d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:36:19.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.193+0000 7f81da7d6640 10 monclient(hunting): picked mon.noname-a con 0x7f81d40b4e70 addr v1:172.21.15.3:6789/0 2024-03-25T18:36:19.195 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.193+0000 7f81da7d6640 10 monclient(hunting): picked mon.noname-c con 0x7f81d40b44b0 addr v1:172.21.15.16:6789/0 2024-03-25T18:36:19.195 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.193+0000 7f81da7d6640 10 monclient(hunting): picked mon.noname-b con 0x7f81d40b1b00 addr v1:172.21.15.3:6790/0 2024-03-25T18:36:19.195 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.193+0000 7f81da7d6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81d40a96f0 con 0x7f81d40b4e70 2024-03-25T18:36:19.195 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.193+0000 7f81da7d6640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81d40ab0e0 con 0x7f81d40b44b0 2024-03-25T18:36:19.195 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.193+0000 7f81da7d6640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81d4145f70 con 0x7f81d40b1b00 2024-03-25T18:36:19.195 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.193+0000 7f81da7d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:19.195 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.193+0000 7f81da7d6640 10 monclient(hunting): authenticate will time out at 8033.453804s 2024-03-25T18:36:19.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.199+0000 7f81d3fff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f81d40b4e70 0x7f81d413f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:46126/0 (socket says 172.21.15.3:46126) 2024-03-25T18:36:19.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.199+0000 7f81d3fff640 1 -- 172.21.15.3:0/1770302027 learned_addr learned my addr 172.21.15.3:0/1770302027 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:36:19.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.211+0000 7f81d2ffd640 1 -- 172.21.15.3:0/1770302027 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3591915309 0 0) 0x7f81d40a96f0 con 0x7f81d40b4e70 2024-03-25T18:36:19.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.211+0000 7f81d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:19.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.211+0000 7f81d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:19.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.211+0000 7f81d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:19.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.211+0000 7f81d2ffd640 10 monclient(hunting): my global_id is 4740 2024-03-25T18:36:19.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.211+0000 7f81d2ffd640 1 -- 172.21.15.3:0/1770302027 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81bc003520 con 0x7f81d40b4e70 2024-03-25T18:36:19.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.227+0000 7f81d2ffd640 1 -- 172.21.15.3:0/1770302027 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4167792460 0 0) 0x7f81bc003520 con 0x7f81d40b4e70 2024-03-25T18:36:19.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.227+0000 7f81d2ffd640 1 -- 172.21.15.3:0/1770302027 >> v1:172.21.15.3:6790/0 conn(0x7f81d40b1b00 legacy=0x7f81d414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:19.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.227+0000 7f81d2ffd640 1 -- 172.21.15.3:0/1770302027 >> v1:172.21.15.16:6789/0 conn(0x7f81d40b44b0 legacy=0x7f81d4145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:19.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.227+0000 7f81d2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:36:19.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.227+0000 7f81d2ffd640 1 monclient: found mon.noname-a 2024-03-25T18:36:19.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.227+0000 7f81d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:19.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.227+0000 7f81d2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:36:19.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.227+0000 7f81d2ffd640 1 -- 172.21.15.3:0/1770302027 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d40a9b80 con 0x7f81d40b4e70 2024-03-25T18:36:19.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.227+0000 7f81d2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:36:19.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.227+0000 7f81d2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:36:19.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.227+0000 7f81d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:19.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.227+0000 7f81d2ffd640 1 -- 172.21.15.3:0/1770302027 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f81c4001690 con 0x7f81d40b4e70 2024-03-25T18:36:19.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.227+0000 7f81d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:19.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.227+0000 7f81d2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:36:19.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.227+0000 7f81d2ffd640 10 monclient: dump: 2024-03-25T18:36:19.229 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:36:19.229 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:36:19.229 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:19.229 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:19.229 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:19.230 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:36:19.230 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:36:19.230 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:36:19.230 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:36:19.230 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:36:19.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.227+0000 7f81da7d6640 5 monclient: authenticate success, global_id 4740 2024-03-25T18:36:19.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.227+0000 7f81da7d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:19.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.242+0000 7f81d2ffd640 1 -- 172.21.15.3:0/1770302027 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81c40017f0 con 0x7f81d40b4e70 2024-03-25T18:36:19.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.242+0000 7f81d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:19.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.242+0000 7f81d2ffd640 1 -- 172.21.15.3:0/1770302027 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f81c4001c00 con 0x7f81d40b4e70 2024-03-25T18:36:19.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.242+0000 7f81d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:19.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.242+0000 7f81d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:19.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.242+0000 7f81d2ffd640 10 monclient: dump: 2024-03-25T18:36:19.243 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:36:19.243 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:36:19.243 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:19.243 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:19.243 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:19.243 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:36:19.243 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:36:19.244 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:36:19.244 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:36:19.244 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:36:19.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.242+0000 7f81da7d6640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:19.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.242+0000 7f81da7d6640 4 set_mon_vals no callback set 2024-03-25T18:36:19.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.243+0000 7f81da7d6640 10 monclient: shutdown 2024-03-25T18:36:19.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.243+0000 7f81da7d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:19.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.243+0000 7f81da7d6640 1 -- 172.21.15.3:0/1770302027 >> v1:172.21.15.3:6789/0 conn(0x7f81d40b4e70 legacy=0x7f81d413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:19.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.243+0000 7f81da7d6640 1 -- 172.21.15.3:0/1770302027 shutdown_connections 2024-03-25T18:36:19.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.243+0000 7f81da7d6640 1 -- 172.21.15.3:0/1770302027 >> 172.21.15.3:0/1770302027 conn(0x7f81d40617a0 msgr2=0x7f81d4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:19.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.243+0000 7f81da7d6640 1 -- 172.21.15.3:0/1770302027 shutdown_connections 2024-03-25T18:36:19.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.243+0000 7f81da7d6640 1 -- 172.21.15.3:0/1770302027 wait complete. 2024-03-25T18:36:19.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.244+0000 7f81da7d6640 10 monclient: build_initial_monmap 2024-03-25T18:36:19.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.244+0000 7f81da7d6640 10 monclient: monmap: 2024-03-25T18:36:19.245 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:36:19.245 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:19.245 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:36:19.245 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:36:19.246 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:19.246 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:36:19.246 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:36:19.246 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:36:19.246 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:36:19.246 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:36:19.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.244+0000 7f81da7d6640 1 librados: starting msgr at 2024-03-25T18:36:19.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.244+0000 7f81da7d6640 1 librados: starting objecter 2024-03-25T18:36:19.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.245+0000 7f81da7d6640 1 Processor -- start 2024-03-25T18:36:19.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.245+0000 7f81da7d6640 1 -- start start 2024-03-25T18:36:19.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.245+0000 7f81da7d6640 1 librados: setting wanted keys 2024-03-25T18:36:19.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.245+0000 7f81da7d6640 1 librados: calling monclient init 2024-03-25T18:36:19.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.245+0000 7f81da7d6640 10 monclient: init 2024-03-25T18:36:19.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.245+0000 7f81da7d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:19.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.245+0000 7f81da7d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:36:19.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.246+0000 7f81da7d6640 10 monclient(hunting): picked mon.noname-a con 0x7f81d40b1b00 addr v1:172.21.15.3:6789/0 2024-03-25T18:36:19.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.246+0000 7f81da7d6640 10 monclient(hunting): picked mon.noname-c con 0x7f81d40b44b0 addr v1:172.21.15.3:6790/0 2024-03-25T18:36:19.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.246+0000 7f81da7d6640 10 monclient(hunting): picked mon.noname-b con 0x7f81d40b4e70 addr v1:172.21.15.16:6789/0 2024-03-25T18:36:19.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.246+0000 7f81da7d6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81d41451c0 con 0x7f81d40b1b00 2024-03-25T18:36:19.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.246+0000 7f81da7d6640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81d413b6e0 con 0x7f81d40b4e70 2024-03-25T18:36:19.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.246+0000 7f81da7d6640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81d413cb50 con 0x7f81d40b44b0 2024-03-25T18:36:19.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.246+0000 7f81da7d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:19.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.246+0000 7f81da7d6640 10 monclient(hunting): authenticate will time out at 8033.506563s 2024-03-25T18:36:19.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.251+0000 7f81d3fff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f81d40b1b00 0x7f81d413aa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:46172/0 (socket says 172.21.15.3:46172) 2024-03-25T18:36:19.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.251+0000 7f81d3fff640 1 -- 172.21.15.3:0/391264807 learned_addr learned my addr 172.21.15.3:0/391264807 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:36:19.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.253+0000 7f81d0ff9640 10 client.?.objecter ms_handle_connect 0x7f81d40b1b00 2024-03-25T18:36:19.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.253+0000 7f81d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:19.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.256+0000 7f81d0ff9640 10 client.?.objecter ms_handle_connect 0x7f81d40b44b0 2024-03-25T18:36:19.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.256+0000 7f81d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:19.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.263+0000 7f81d0ff9640 1 -- 172.21.15.3:0/391264807 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 105500497 0 0) 0x7f81d41451c0 con 0x7f81d40b1b00 2024-03-25T18:36:19.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.263+0000 7f81d0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:36:19.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.263+0000 7f81d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:19.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.263+0000 7f81d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:19.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.263+0000 7f81d0ff9640 10 monclient(hunting): my global_id is 4743 2024-03-25T18:36:19.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.263+0000 7f81d0ff9640 1 -- 172.21.15.3:0/391264807 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81c0004eb0 con 0x7f81d40b1b00 2024-03-25T18:36:19.268 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.267+0000 7f81d0ff9640 10 client.?.objecter ms_handle_connect 0x7f81d40b4e70 2024-03-25T18:36:19.268 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.267+0000 7f81d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:19.274 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.273+0000 7f81d0ff9640 1 -- 172.21.15.3:0/391264807 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2372832942 0 0) 0x7f81d413cb50 con 0x7f81d40b44b0 2024-03-25T18:36:19.274 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.273+0000 7f81d0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:36:19.274 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.273+0000 7f81d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:19.274 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.273+0000 7f81d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:19.274 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.273+0000 7f81d0ff9640 10 monclient(hunting): my global_id is 4685 2024-03-25T18:36:19.274 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.273+0000 7f81d0ff9640 1 -- 172.21.15.3:0/391264807 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81d41451c0 con 0x7f81d40b44b0 2024-03-25T18:36:19.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.278+0000 7f81d0ff9640 1 -- 172.21.15.3:0/391264807 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 478774147 0 0) 0x7f81c0004eb0 con 0x7f81d40b1b00 2024-03-25T18:36:19.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.279+0000 7f81d0ff9640 1 -- 172.21.15.3:0/391264807 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81d413cb50 con 0x7f81d40b1b00 2024-03-25T18:36:19.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.279+0000 7f81d0ff9640 1 -- 172.21.15.3:0/391264807 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f81c4001c00 con 0x7f81d40b1b00 2024-03-25T18:36:19.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.279+0000 7f81d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:19.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.279+0000 7f81d0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:36:19.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.279+0000 7f81d0ff9640 10 monclient(hunting): dump: 2024-03-25T18:36:19.280 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:36:19.280 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:36:19.280 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:19.280 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:19.280 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:19.281 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:36:19.281 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:36:19.281 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:36:19.281 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:36:19.281 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:36:19.281 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.279+0000 7f81d0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:19.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.285+0000 7f81d0ff9640 1 -- 172.21.15.3:0/391264807 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4109486848 0 0) 0x7f81d413cb50 con 0x7f81d40b1b00 2024-03-25T18:36:19.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.285+0000 7f81d0ff9640 1 -- 172.21.15.3:0/391264807 >> v1:172.21.15.3:6790/0 conn(0x7f81d40b44b0 legacy=0x7f81d4143b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:19.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.285+0000 7f81d0ff9640 1 -- 172.21.15.3:0/391264807 >> v1:172.21.15.16:6789/0 conn(0x7f81d40b4e70 legacy=0x7f81d4142790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:19.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.285+0000 7f81d0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:36:19.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.285+0000 7f81d0ff9640 1 monclient: found mon.a 2024-03-25T18:36:19.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.285+0000 7f81d0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:19.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.285+0000 7f81d0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:36:19.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.285+0000 7f81d0ff9640 1 -- 172.21.15.3:0/391264807 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d4139f50 con 0x7f81d40b1b00 2024-03-25T18:36:19.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.285+0000 7f81d0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:36:19.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.285+0000 7f81d0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:36:19.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.285+0000 7f81d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:19.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.285+0000 7f81da7d6640 5 monclient: authenticate success, global_id 4743 2024-03-25T18:36:19.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.285+0000 7f81da7d6640 10 monclient: _renew_subs 2024-03-25T18:36:19.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.285+0000 7f81da7d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:36:19.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.285+0000 7f81da7d6640 1 -- 172.21.15.3:0/391264807 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81d413bb70 con 0x7f81d40b1b00 2024-03-25T18:36:19.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.285+0000 7f81da7d6640 10 client.4743.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:19.288 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.285+0000 7f81da7d6640 10 monclient: _renew_subs 2024-03-25T18:36:19.288 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.285+0000 7f81da7d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:36:19.288 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.285+0000 7f81da7d6640 1 -- 172.21.15.3:0/391264807 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81d4144210 con 0x7f81d40b1b00 2024-03-25T18:36:19.288 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.285+0000 7f81da7d6640 1 librados: init done 2024-03-25T18:36:19.288 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.286+0000 7f81da7d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:19.288 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.286+0000 7f81da7d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:19.289 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.286+0000 7f81da7d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:36:19.289 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.286+0000 7f81da7d6640 1 -- 172.21.15.3:0/391264807 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81d413b450 con 0x7f81d40b1b00 2024-03-25T18:36:19.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.605+0000 7f81d0ff9640 1 -- 172.21.15.3:0/391264807 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81c4004a90 con 0x7f81d40b1b00 2024-03-25T18:36:19.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.605+0000 7f81d0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:19.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.605+0000 7f81d0ff9640 1 -- 172.21.15.3:0/391264807 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f81c4004da0 con 0x7f81d40b1b00 2024-03-25T18:36:19.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.605+0000 7f81d0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:19.607 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.605+0000 7f81d0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:19.607 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.605+0000 7f81d0ff9640 10 monclient: dump: 2024-03-25T18:36:19.607 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:36:19.607 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:36:19.607 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:19.607 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:19.607 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:19.607 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:36:19.607 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:36:19.607 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:36:19.607 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:36:19.607 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:36:19.607 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.605+0000 7f81d27fc640 4 set_mon_vals no callback set 2024-03-25T18:36:19.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.606+0000 7f81d0ff9640 1 -- 172.21.15.3:0/391264807 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f81c401ad70 con 0x7f81d40b1b00 2024-03-25T18:36:19.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.606+0000 7f81d37fe640 10 monclient: get_auth_request con 0x7f81c006dcc0 auth_method 0 2024-03-25T18:36:19.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.607+0000 7f81d0ff9640 10 client.4743.objecter ms_handle_connect 0x7f81c006dcc0 2024-03-25T18:36:19.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.612+0000 7f81d0ff9640 1 -- 172.21.15.3:0/391264807 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2800263165 0 0) 0x7f81c4054910 con 0x7f81d40b1b00 2024-03-25T18:36:19.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.612+0000 7f81d0ff9640 10 client.4743.objecter ms_dispatch 0x7f81d4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:36:19.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.612+0000 7f81d0ff9640 3 client.4743.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:36:19.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.612+0000 7f81d0ff9640 3 client.4743.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:36:19.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.612+0000 7f81d0ff9640 20 client.4743.objecter dump_active .. 0 homeless 2024-03-25T18:36:19.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.612+0000 7f81d0ff9640 1 -- 172.21.15.3:0/391264807 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f81c4050250 con 0x7f81d40b1b00 2024-03-25T18:36:19.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.612+0000 7f81d0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:19.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.612+0000 7f81d0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:19.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.874+0000 7f81da7d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:19.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.874+0000 7f81da7d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:19.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.874+0000 7f81da7d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:36:19.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.874+0000 7f81da7d6640 1 -- 172.21.15.3:0/391264807 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81d404f730 con 0x7f81d40b1b00 2024-03-25T18:36:19.909 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.908+0000 7f81d0ff9640 1 -- 172.21.15.3:0/391264807 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 716832884) 0x7f81c4052350 con 0x7f81d40b1b00 2024-03-25T18:36:19.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.908+0000 7f81d0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:19.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.908+0000 7f81d0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:19.910 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:36:19.910 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":11,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:35:08.249841+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:35:08.249841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:19.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.910+0000 7f81da7d6640 10 librados: watch_flush enter 2024-03-25T18:36:19.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.911+0000 7f81da7d6640 10 librados: watch_flush exit 2024-03-25T18:36:19.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.911+0000 7f81da7d6640 20 client.4743.objecter shutdown clearing up homeless session... 2024-03-25T18:36:19.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.911+0000 7f81da7d6640 10 client.4743.objecter successfully canceled tick 2024-03-25T18:36:19.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.911+0000 7f81da7d6640 1 -- 172.21.15.3:0/391264807 >> v1:172.21.15.3:6800/4195140966 conn(0x7f81c006dcc0 legacy=0x7f81c0070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:19.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.911+0000 7f81da7d6640 10 monclient: shutdown 2024-03-25T18:36:19.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.911+0000 7f81da7d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:19.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.911+0000 7f81da7d6640 1 -- 172.21.15.3:0/391264807 >> v1:172.21.15.3:6789/0 conn(0x7f81d40b1b00 legacy=0x7f81d413aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:19.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.911+0000 7f81da7d6640 1 -- 172.21.15.3:0/391264807 shutdown_connections 2024-03-25T18:36:19.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.911+0000 7f81da7d6640 1 -- 172.21.15.3:0/391264807 >> 172.21.15.3:0/391264807 conn(0x7f81d40617a0 msgr2=0x7f81d40b6690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:19.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.911+0000 7f81da7d6640 1 -- 172.21.15.3:0/391264807 shutdown_connections 2024-03-25T18:36:19.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.911+0000 7f81da7d6640 1 -- 172.21.15.3:0/391264807 wait complete. 2024-03-25T18:36:19.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:19.911+0000 7f81da7d6640 1 librados: shutdown 2024-03-25T18:36:19.923 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:23.626 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:11:37.355 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:23.632 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:11:37.361 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:23.663 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:11:37.392 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:23.684 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:11:37.376 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:23.687 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:11:37.379 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:23.702 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:11:37.431 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:23.706 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:11:37.435 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:23.721 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:11:37.413 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:23.736 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:11:37.465 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:26.730 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:26.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.841+0000 7fd26d1e2640 10 monclient: get_monmap_and_config 2024-03-25T18:36:26.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.841+0000 7fd26d1e2640 10 monclient: build_initial_monmap 2024-03-25T18:36:26.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.841+0000 7fd26d1e2640 10 monclient: monmap: 2024-03-25T18:36:26.842 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:36:26.842 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:36:26.843 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:36:26.842641+0000 2024-03-25T18:36:26.843 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:36:26.842641+0000 2024-03-25T18:36:26.843 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:26.843 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:36:26.843 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:36:26.843 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:36:26.843 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:36:26.843 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:36:26.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.842+0000 7fd26d1e2640 1 Processor -- start 2024-03-25T18:36:26.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.842+0000 7fd26d1e2640 1 -- start start 2024-03-25T18:36:26.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.842+0000 7fd26d1e2640 10 monclient: init 2024-03-25T18:36:26.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.843+0000 7fd26d1e2640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:26.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.843+0000 7fd26d1e2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:36:26.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.843+0000 7fd26d1e2640 10 monclient(hunting): picked mon.noname-b con 0x7fd2680b4ff0 addr v1:172.21.15.3:6790/0 2024-03-25T18:36:26.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.843+0000 7fd26d1e2640 10 monclient(hunting): picked mon.noname-c con 0x7fd2680b4670 addr v1:172.21.15.16:6789/0 2024-03-25T18:36:26.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.843+0000 7fd26d1e2640 10 monclient(hunting): picked mon.noname-a con 0x7fd26813c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:36:26.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.843+0000 7fd26d1e2640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2680ab0e0 con 0x7fd26813c480 2024-03-25T18:36:26.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.843+0000 7fd26d1e2640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2680ab3c0 con 0x7fd2680b4670 2024-03-25T18:36:26.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.843+0000 7fd26d1e2640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2680a9f30 con 0x7fd2680b4ff0 2024-03-25T18:36:26.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.843+0000 7fd26d1e2640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:26.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.843+0000 7fd26d1e2640 10 monclient(hunting): authenticate will time out at 8041.103559s 2024-03-25T18:36:26.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.846+0000 7fd266d9d640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fd2680b4ff0 0x7fd26813bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:55256/0 (socket says 172.21.15.3:55256) 2024-03-25T18:36:26.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.846+0000 7fd266d9d640 1 -- 172.21.15.3:0/3982215245 learned_addr learned my addr 172.21.15.3:0/3982215245 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:36:26.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.920+0000 7fd265d9b640 1 -- 172.21.15.3:0/3982215245 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299567620 0 0) 0x7fd2680ab0e0 con 0x7fd26813c480 2024-03-25T18:36:26.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.921+0000 7fd265d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:36:26.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.921+0000 7fd265d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:26.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.921+0000 7fd265d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:26.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.921+0000 7fd265d9b640 10 monclient(hunting): my global_id is 4746 2024-03-25T18:36:26.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.921+0000 7fd265d9b640 1 -- 172.21.15.3:0/3982215245 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd250003520 con 0x7fd26813c480 2024-03-25T18:36:26.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.931+0000 7fd265d9b640 1 -- 172.21.15.3:0/3982215245 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 715604543 0 0) 0x7fd2680ab3c0 con 0x7fd2680b4670 2024-03-25T18:36:26.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.932+0000 7fd265d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:36:26.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.932+0000 7fd265d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:26.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.932+0000 7fd265d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:26.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.932+0000 7fd265d9b640 10 monclient(hunting): my global_id is 4684 2024-03-25T18:36:26.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.932+0000 7fd265d9b640 1 -- 172.21.15.3:0/3982215245 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2680ab0e0 con 0x7fd2680b4670 2024-03-25T18:36:26.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.936+0000 7fd265d9b640 1 -- 172.21.15.3:0/3982215245 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2579557208 0 0) 0x7fd250003520 con 0x7fd26813c480 2024-03-25T18:36:26.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.936+0000 7fd265d9b640 1 -- 172.21.15.3:0/3982215245 >> v1:172.21.15.3:6790/0 conn(0x7fd2680b4ff0 legacy=0x7fd26813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:26.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.936+0000 7fd265d9b640 1 -- 172.21.15.3:0/3982215245 >> v1:172.21.15.16:6789/0 conn(0x7fd2680b4670 legacy=0x7fd2681425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:26.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.936+0000 7fd265d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:36:26.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.936+0000 7fd265d9b640 1 monclient: found mon.noname-a 2024-03-25T18:36:26.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.936+0000 7fd265d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:26.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.936+0000 7fd265d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-25T18:36:26.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.936+0000 7fd265d9b640 1 -- 172.21.15.3:0/3982215245 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2680a9b70 con 0x7fd26813c480 2024-03-25T18:36:26.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.936+0000 7fd265d9b640 10 monclient: _finish_auth 0 2024-03-25T18:36:26.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.936+0000 7fd265d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:36:26.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.936+0000 7fd265d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:26.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.936+0000 7fd265d9b640 1 -- 172.21.15.3:0/3982215245 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd260001a10 con 0x7fd26813c480 2024-03-25T18:36:26.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.936+0000 7fd26d1e2640 5 monclient: authenticate success, global_id 4746 2024-03-25T18:36:26.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.936+0000 7fd26d1e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:26.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.936+0000 7fd265d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:26.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.936+0000 7fd265d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:36:26.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.936+0000 7fd265d9b640 10 monclient: dump: 2024-03-25T18:36:26.939 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:36:26.939 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:36:26.939 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:26.939 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:26.939 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:26.939 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:36:26.940 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:36:26.940 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:36:26.940 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:36:26.940 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:36:26.940 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.936+0000 7fd26d1e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:26.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.952+0000 7fd265d9b640 1 -- 172.21.15.3:0/3982215245 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd260001b70 con 0x7fd26813c480 2024-03-25T18:36:26.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.952+0000 7fd265d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:26.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.952+0000 7fd265d9b640 1 -- 172.21.15.3:0/3982215245 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd260001f80 con 0x7fd26813c480 2024-03-25T18:36:26.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.952+0000 7fd265d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:26.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.952+0000 7fd265d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:26.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.952+0000 7fd265d9b640 10 monclient: dump: 2024-03-25T18:36:26.954 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:36:26.954 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:36:26.954 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:26.954 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:26.954 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:26.955 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:36:26.955 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:36:26.955 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:36:26.955 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:36:26.955 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:36:26.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.953+0000 7fd26d1e2640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:26.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.953+0000 7fd26d1e2640 4 set_mon_vals no callback set 2024-03-25T18:36:26.956 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.953+0000 7fd26d1e2640 10 monclient: shutdown 2024-03-25T18:36:26.956 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.953+0000 7fd26d1e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:26.956 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.953+0000 7fd26d1e2640 1 -- 172.21.15.3:0/3982215245 >> v1:172.21.15.3:6789/0 conn(0x7fd26813c480 legacy=0x7fd26814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:26.956 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.954+0000 7fd26d1e2640 1 -- 172.21.15.3:0/3982215245 shutdown_connections 2024-03-25T18:36:26.956 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.954+0000 7fd26d1e2640 1 -- 172.21.15.3:0/3982215245 >> 172.21.15.3:0/3982215245 conn(0x7fd2680617a0 msgr2=0x7fd268061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:26.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.954+0000 7fd26d1e2640 1 -- 172.21.15.3:0/3982215245 shutdown_connections 2024-03-25T18:36:26.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.954+0000 7fd26d1e2640 1 -- 172.21.15.3:0/3982215245 wait complete. 2024-03-25T18:36:26.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.955+0000 7fd26d1e2640 10 monclient: build_initial_monmap 2024-03-25T18:36:26.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.955+0000 7fd26d1e2640 10 monclient: monmap: 2024-03-25T18:36:26.957 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:36:26.957 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:26.957 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:36:26.957 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:36:26.957 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:26.957 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:36:26.957 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:36:26.957 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:36:26.957 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:36:26.958 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:36:26.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.955+0000 7fd26d1e2640 1 librados: starting msgr at 2024-03-25T18:36:26.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.955+0000 7fd26d1e2640 1 librados: starting objecter 2024-03-25T18:36:26.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.955+0000 7fd26d1e2640 1 Processor -- start 2024-03-25T18:36:26.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.955+0000 7fd26d1e2640 1 -- start start 2024-03-25T18:36:26.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.955+0000 7fd26d1e2640 1 librados: setting wanted keys 2024-03-25T18:36:26.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.955+0000 7fd26d1e2640 1 librados: calling monclient init 2024-03-25T18:36:26.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.955+0000 7fd26d1e2640 10 monclient: init 2024-03-25T18:36:26.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.956+0000 7fd26d1e2640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:26.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.956+0000 7fd26d1e2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:36:26.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.956+0000 7fd26d1e2640 10 monclient(hunting): picked mon.noname-c con 0x7fd2680b4670 addr v1:172.21.15.3:6790/0 2024-03-25T18:36:26.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.956+0000 7fd26d1e2640 10 monclient(hunting): picked mon.noname-b con 0x7fd2680b4ff0 addr v1:172.21.15.16:6789/0 2024-03-25T18:36:26.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.956+0000 7fd26d1e2640 10 monclient(hunting): picked mon.noname-a con 0x7fd26813c480 addr v1:172.21.15.3:6789/0 2024-03-25T18:36:26.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.956+0000 7fd26d1e2640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd268146240 con 0x7fd26813c480 2024-03-25T18:36:26.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.956+0000 7fd26d1e2640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd268144770 con 0x7fd2680b4ff0 2024-03-25T18:36:26.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.956+0000 7fd26d1e2640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd268144a50 con 0x7fd2680b4670 2024-03-25T18:36:26.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.956+0000 7fd26d1e2640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:26.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.956+0000 7fd26d1e2640 10 monclient(hunting): authenticate will time out at 8041.216867s 2024-03-25T18:36:26.971 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.970+0000 7fd26659c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd2680b4ff0 0x7fd26813d0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:58982/0 (socket says 172.21.15.3:58982) 2024-03-25T18:36:26.971 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.970+0000 7fd26659c640 1 -- 172.21.15.3:0/1963054206 learned_addr learned my addr 172.21.15.3:0/1963054206 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:36:26.972 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.971+0000 7fd24f7fe640 10 client.?.objecter ms_handle_connect 0x7fd2680b4ff0 2024-03-25T18:36:26.972 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.971+0000 7fd24f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:26.974 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.974+0000 7fd24f7fe640 10 client.?.objecter ms_handle_connect 0x7fd2680b4670 2024-03-25T18:36:26.974 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.974+0000 7fd24f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:26.980 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.979+0000 7fd24f7fe640 10 client.?.objecter ms_handle_connect 0x7fd26813c480 2024-03-25T18:36:26.980 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.979+0000 7fd24f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:26.980 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.979+0000 7fd24f7fe640 1 -- 172.21.15.3:0/1963054206 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1058912357 0 0) 0x7fd268144770 con 0x7fd2680b4ff0 2024-03-25T18:36:26.980 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.979+0000 7fd24f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:36:26.980 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.979+0000 7fd24f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:26.980 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.979+0000 7fd24f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:26.980 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.979+0000 7fd24f7fe640 10 monclient(hunting): my global_id is 4687 2024-03-25T18:36:26.981 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.979+0000 7fd24f7fe640 1 -- 172.21.15.3:0/1963054206 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd254004eb0 con 0x7fd2680b4ff0 2024-03-25T18:36:26.983 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.982+0000 7fd24f7fe640 1 -- 172.21.15.3:0/1963054206 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 562566897 0 0) 0x7fd268144a50 con 0x7fd2680b4670 2024-03-25T18:36:26.983 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.982+0000 7fd24f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:36:26.984 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.982+0000 7fd24f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:26.984 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.982+0000 7fd24f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:26.984 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.982+0000 7fd24f7fe640 10 monclient(hunting): my global_id is 4691 2024-03-25T18:36:26.984 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.982+0000 7fd24f7fe640 1 -- 172.21.15.3:0/1963054206 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd268144770 con 0x7fd2680b4670 2024-03-25T18:36:26.996 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.994+0000 7fd24f7fe640 1 -- 172.21.15.3:0/1963054206 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 11230154 0 0) 0x7fd254004eb0 con 0x7fd2680b4ff0 2024-03-25T18:36:26.996 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.995+0000 7fd24f7fe640 1 -- 172.21.15.3:0/1963054206 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd268144a50 con 0x7fd2680b4ff0 2024-03-25T18:36:26.996 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.995+0000 7fd24f7fe640 1 -- 172.21.15.3:0/1963054206 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd25c001a10 con 0x7fd2680b4ff0 2024-03-25T18:36:26.996 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.995+0000 7fd24f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:26.996 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.995+0000 7fd24f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:36:26.996 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.995+0000 7fd24f7fe640 10 monclient(hunting): dump: 2024-03-25T18:36:26.996 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:36:26.996 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:36:26.996 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:26.996 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:26.996 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:26.996 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:36:26.997 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:36:26.997 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:36:26.997 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:36:26.997 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:36:26.997 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:26.995+0000 7fd24f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:27.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.004+0000 7fd24f7fe640 1 -- 172.21.15.3:0/1963054206 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3301259563 0 0) 0x7fd268146240 con 0x7fd26813c480 2024-03-25T18:36:27.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.004+0000 7fd24f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:36:27.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.004+0000 7fd24f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:27.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.004+0000 7fd24f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:27.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.004+0000 7fd24f7fe640 10 monclient(hunting): my global_id is 4749 2024-03-25T18:36:27.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.004+0000 7fd24f7fe640 1 -- 172.21.15.3:0/1963054206 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd254004eb0 con 0x7fd26813c480 2024-03-25T18:36:27.012 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.010+0000 7fd24f7fe640 1 -- 172.21.15.3:0/1963054206 <== mon.1 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 145465288 0 0) 0x7fd268144a50 con 0x7fd2680b4ff0 2024-03-25T18:36:27.012 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.010+0000 7fd24f7fe640 1 -- 172.21.15.3:0/1963054206 >> v1:172.21.15.3:6790/0 conn(0x7fd2680b4670 legacy=0x7fd26813c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:27.012 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.010+0000 7fd24f7fe640 1 -- 172.21.15.3:0/1963054206 >> v1:172.21.15.3:6789/0 conn(0x7fd26813c480 legacy=0x7fd2681e2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:27.012 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.010+0000 7fd24f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:36:27.012 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.010+0000 7fd24f7fe640 1 monclient: found mon.b 2024-03-25T18:36:27.012 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.010+0000 7fd24f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:27.012 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.010+0000 7fd24f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:36:27.012 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.010+0000 7fd24f7fe640 1 -- 172.21.15.3:0/1963054206 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd26813c020 con 0x7fd2680b4ff0 2024-03-25T18:36:27.012 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.010+0000 7fd24f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:36:27.012 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.010+0000 7fd24f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:36:27.012 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.010+0000 7fd24f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:27.012 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.010+0000 7fd26d1e2640 5 monclient: authenticate success, global_id 4687 2024-03-25T18:36:27.012 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.010+0000 7fd26d1e2640 10 monclient: _renew_subs 2024-03-25T18:36:27.013 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.011+0000 7fd26d1e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:36:27.013 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.011+0000 7fd26d1e2640 1 -- 172.21.15.3:0/1963054206 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd26813c2d0 con 0x7fd2680b4ff0 2024-03-25T18:36:27.013 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.011+0000 7fd26d1e2640 10 client.4687.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:27.013 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.011+0000 7fd26d1e2640 10 monclient: _renew_subs 2024-03-25T18:36:27.013 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.011+0000 7fd26d1e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:36:27.013 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.011+0000 7fd26d1e2640 1 -- 172.21.15.3:0/1963054206 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd268143ab0 con 0x7fd2680b4ff0 2024-03-25T18:36:27.013 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.011+0000 7fd26d1e2640 1 librados: init done 2024-03-25T18:36:27.013 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.012+0000 7fd26d1e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:27.014 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.012+0000 7fd26d1e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:27.014 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.012+0000 7fd26d1e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:36:27.014 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.012+0000 7fd26d1e2640 1 -- 172.21.15.3:0/1963054206 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd26806dbd0 con 0x7fd2680b4ff0 2024-03-25T18:36:27.306 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.305+0000 7fd24f7fe640 1 -- 172.21.15.3:0/1963054206 <== mon.1 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd25c001c50 con 0x7fd2680b4ff0 2024-03-25T18:36:27.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.305+0000 7fd24f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:27.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.305+0000 7fd24f7fe640 1 -- 172.21.15.3:0/1963054206 <== mon.1 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7fd25c003a60 con 0x7fd2680b4ff0 2024-03-25T18:36:27.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.305+0000 7fd24f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:27.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.305+0000 7fd24f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:27.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.305+0000 7fd24f7fe640 10 monclient: dump: 2024-03-25T18:36:27.307 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:36:27.307 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:36:27.307 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:27.307 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:27.307 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:27.307 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:36:27.307 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:36:27.307 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:36:27.308 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:36:27.308 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:36:27.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.305+0000 7fd26559a640 4 set_mon_vals no callback set 2024-03-25T18:36:27.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.306+0000 7fd24f7fe640 1 -- 172.21.15.3:0/1963054206 <== mon.1 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7fd25c003c80 con 0x7fd2680b4ff0 2024-03-25T18:36:27.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.307+0000 7fd266d9d640 10 monclient: get_auth_request con 0x7fd25406de00 auth_method 0 2024-03-25T18:36:27.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.307+0000 7fd24f7fe640 10 client.4687.objecter ms_handle_connect 0x7fd25406de00 2024-03-25T18:36:27.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.312+0000 7fd24f7fe640 1 -- 172.21.15.3:0/1963054206 <== mon.1 v1:172.21.15.16:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2800263165 0 0) 0x7fd25c054890 con 0x7fd2680b4ff0 2024-03-25T18:36:27.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.312+0000 7fd24f7fe640 10 client.4687.objecter ms_dispatch 0x7fd268000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:36:27.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.312+0000 7fd24f7fe640 3 client.4687.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:36:27.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.312+0000 7fd24f7fe640 3 client.4687.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:36:27.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.312+0000 7fd24f7fe640 20 client.4687.objecter dump_active .. 0 homeless 2024-03-25T18:36:27.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.312+0000 7fd24f7fe640 1 -- 172.21.15.3:0/1963054206 <== mon.1 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd25c0503a0 con 0x7fd2680b4ff0 2024-03-25T18:36:27.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.312+0000 7fd24f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:27.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.312+0000 7fd24f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:27.585 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.584+0000 7fd26d1e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:27.585 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.584+0000 7fd26d1e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:27.585 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.584+0000 7fd26d1e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.16:6789/0 2024-03-25T18:36:27.585 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.584+0000 7fd26d1e2640 1 -- 172.21.15.3:0/1963054206 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd268146640 con 0x7fd2680b4ff0 2024-03-25T18:36:27.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.608+0000 7fd24f7fe640 1 -- 172.21.15.3:0/1963054206 <== mon.1 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 716832884) 0x7fd25c0522d0 con 0x7fd2680b4ff0 2024-03-25T18:36:27.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.609+0000 7fd24f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:27.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.609+0000 7fd24f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:27.610 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:36:27.610 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":11,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:35:08.249841+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:35:08.249841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:27.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.611+0000 7fd26d1e2640 10 librados: watch_flush enter 2024-03-25T18:36:27.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.611+0000 7fd26d1e2640 10 librados: watch_flush exit 2024-03-25T18:36:27.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.611+0000 7fd26d1e2640 20 client.4687.objecter shutdown clearing up homeless session... 2024-03-25T18:36:27.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.611+0000 7fd26d1e2640 10 client.4687.objecter successfully canceled tick 2024-03-25T18:36:27.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.611+0000 7fd26d1e2640 1 -- 172.21.15.3:0/1963054206 >> v1:172.21.15.3:6800/4195140966 conn(0x7fd25406de00 legacy=0x7fd2540702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:27.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.611+0000 7fd26d1e2640 10 monclient: shutdown 2024-03-25T18:36:27.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.611+0000 7fd26d1e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:27.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.611+0000 7fd26d1e2640 1 -- 172.21.15.3:0/1963054206 >> v1:172.21.15.16:6789/0 conn(0x7fd2680b4ff0 legacy=0x7fd26813d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:27.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.611+0000 7fd26d1e2640 1 -- 172.21.15.3:0/1963054206 shutdown_connections 2024-03-25T18:36:27.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.611+0000 7fd26d1e2640 1 -- 172.21.15.3:0/1963054206 >> 172.21.15.3:0/1963054206 conn(0x7fd2680617a0 msgr2=0x7fd2681407a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:27.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.611+0000 7fd26d1e2640 1 -- 172.21.15.3:0/1963054206 shutdown_connections 2024-03-25T18:36:27.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.611+0000 7fd26d1e2640 1 -- 172.21.15.3:0/1963054206 wait complete. 2024-03-25T18:36:27.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:27.612+0000 7fd26d1e2640 1 librados: shutdown 2024-03-25T18:36:27.624 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:34.429 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:34.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.545+0000 7f63ac4cf640 10 monclient: get_monmap_and_config 2024-03-25T18:36:34.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.545+0000 7f63ac4cf640 10 monclient: build_initial_monmap 2024-03-25T18:36:34.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.545+0000 7f63ac4cf640 10 monclient: monmap: 2024-03-25T18:36:34.546 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:36:34.546 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:36:34.546 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:36:34.546625+0000 2024-03-25T18:36:34.546 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:36:34.546625+0000 2024-03-25T18:36:34.546 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:34.547 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:36:34.547 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:36:34.547 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-c 2024-03-25T18:36:34.547 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-25T18:36:34.547 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:36:34.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.546+0000 7f63ac4cf640 1 Processor -- start 2024-03-25T18:36:34.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.546+0000 7f63ac4cf640 1 -- start start 2024-03-25T18:36:34.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.546+0000 7f63ac4cf640 10 monclient: init 2024-03-25T18:36:34.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.546+0000 7f63ac4cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:34.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.547+0000 7f63ac4cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:36:34.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.547+0000 7f63ac4cf640 10 monclient(hunting): picked mon.noname-c con 0x7f63a40b4e70 addr v1:172.21.15.16:6789/0 2024-03-25T18:36:34.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.547+0000 7f63ac4cf640 10 monclient(hunting): picked mon.noname-a con 0x7f63a40b44b0 addr v1:172.21.15.3:6789/0 2024-03-25T18:36:34.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.547+0000 7f63ac4cf640 10 monclient(hunting): picked mon.noname-b con 0x7f63a40b1b00 addr v1:172.21.15.3:6790/0 2024-03-25T18:36:34.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.547+0000 7f63ac4cf640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a40a96f0 con 0x7f63a40b44b0 2024-03-25T18:36:34.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.547+0000 7f63ac4cf640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a40ab0e0 con 0x7f63a40b4e70 2024-03-25T18:36:34.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.547+0000 7f63ac4cf640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a4145f70 con 0x7f63a40b1b00 2024-03-25T18:36:34.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.547+0000 7f63ac4cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:34.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.547+0000 7f63ac4cf640 10 monclient(hunting): authenticate will time out at 8048.807694s 2024-03-25T18:36:34.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.556+0000 7f63aa26b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f63a40b4e70 0x7f63a413f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:55018/0 (socket says 172.21.15.3:55018) 2024-03-25T18:36:34.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.556+0000 7f63aa26b640 1 -- 172.21.15.3:0/3986143710 learned_addr learned my addr 172.21.15.3:0/3986143710 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:36:34.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.565+0000 7f63a9269640 1 -- 172.21.15.3:0/3986143710 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1036632512 0 0) 0x7f63a40ab0e0 con 0x7f63a40b4e70 2024-03-25T18:36:34.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.565+0000 7f63a9269640 10 monclient(hunting): _negotiate 2024-03-25T18:36:34.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.565+0000 7f63a9269640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:34.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.565+0000 7f63a9269640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:34.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.565+0000 7f63a9269640 10 monclient(hunting): my global_id is 4690 2024-03-25T18:36:34.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.565+0000 7f63a9269640 1 -- 172.21.15.3:0/3986143710 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6390003520 con 0x7f63a40b4e70 2024-03-25T18:36:34.568 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.568+0000 7f63a9269640 1 -- 172.21.15.3:0/3986143710 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 30555077 0 0) 0x7f63a4145f70 con 0x7f63a40b1b00 2024-03-25T18:36:34.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.568+0000 7f63a9269640 10 monclient(hunting): _negotiate 2024-03-25T18:36:34.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.568+0000 7f63a9269640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:34.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.568+0000 7f63a9269640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:34.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.568+0000 7f63a9269640 10 monclient(hunting): my global_id is 4694 2024-03-25T18:36:34.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.568+0000 7f63a9269640 1 -- 172.21.15.3:0/3986143710 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63a40ab0e0 con 0x7f63a40b1b00 2024-03-25T18:36:34.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.569+0000 7f63a9269640 1 -- 172.21.15.3:0/3986143710 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3085009964 0 0) 0x7f63a40a96f0 con 0x7f63a40b44b0 2024-03-25T18:36:34.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.569+0000 7f63a9269640 10 monclient(hunting): _negotiate 2024-03-25T18:36:34.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.569+0000 7f63a9269640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:34.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.569+0000 7f63a9269640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:34.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.569+0000 7f63a9269640 10 monclient(hunting): my global_id is 4752 2024-03-25T18:36:34.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.569+0000 7f63a9269640 1 -- 172.21.15.3:0/3986143710 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63a4145f70 con 0x7f63a40b44b0 2024-03-25T18:36:34.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.575+0000 7f63a9269640 1 -- 172.21.15.3:0/3986143710 <== mon.1 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 510267556 0 0) 0x7f6390003520 con 0x7f63a40b4e70 2024-03-25T18:36:34.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.575+0000 7f63a9269640 1 -- 172.21.15.3:0/3986143710 >> v1:172.21.15.3:6790/0 conn(0x7f63a40b1b00 legacy=0x7f63a414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:34.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.575+0000 7f63a9269640 1 -- 172.21.15.3:0/3986143710 >> v1:172.21.15.3:6789/0 conn(0x7f63a40b44b0 legacy=0x7f63a4145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:34.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.575+0000 7f63a9269640 10 monclient: _finish_hunting 0 2024-03-25T18:36:34.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.575+0000 7f63a9269640 1 monclient: found mon.noname-c 2024-03-25T18:36:34.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.575+0000 7f63a9269640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:34.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.575+0000 7f63a9269640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.16:6789/0 2024-03-25T18:36:34.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.575+0000 7f63a9269640 1 -- 172.21.15.3:0/3986143710 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a40a9b80 con 0x7f63a40b4e70 2024-03-25T18:36:34.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.575+0000 7f63a9269640 10 monclient: _finish_auth 0 2024-03-25T18:36:34.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.575+0000 7f63a9269640 10 monclient: _check_auth_tickets 2024-03-25T18:36:34.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.575+0000 7f63a9269640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:34.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.575+0000 7f63a9269640 1 -- 172.21.15.3:0/3986143710 <== mon.1 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f63a0001690 con 0x7f63a40b4e70 2024-03-25T18:36:34.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.575+0000 7f63ac4cf640 5 monclient: authenticate success, global_id 4690 2024-03-25T18:36:34.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.575+0000 7f63a9269640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:34.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.575+0000 7f63a9269640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:36:34.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.575+0000 7f63a9269640 10 monclient: dump: 2024-03-25T18:36:34.577 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:36:34.577 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:36:34.577 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:34.577 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:34.578 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:34.578 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:36:34.578 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:36:34.578 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:36:34.578 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:36:34.578 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:36:34.578 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.575+0000 7f63ac4cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:34.586 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.585+0000 7f63a9269640 1 -- 172.21.15.3:0/3986143710 <== mon.1 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63a00018b0 con 0x7f63a40b4e70 2024-03-25T18:36:34.586 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.585+0000 7f63a9269640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:34.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.585+0000 7f63ac4cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:34.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.585+0000 7f63a9269640 1 -- 172.21.15.3:0/3986143710 <== mon.1 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f63a0001c40 con 0x7f63a40b4e70 2024-03-25T18:36:34.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.585+0000 7f63ac4cf640 4 set_mon_vals no callback set 2024-03-25T18:36:34.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.585+0000 7f63a9269640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:34.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.585+0000 7f63a9269640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:34.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.585+0000 7f63a9269640 10 monclient: dump: 2024-03-25T18:36:34.587 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:36:34.587 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:36:34.587 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:34.587 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:34.587 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:34.587 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:36:34.587 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:36:34.587 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:36:34.587 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:36:34.587 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:36:34.589 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.586+0000 7f63ac4cf640 10 monclient: shutdown 2024-03-25T18:36:34.589 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.586+0000 7f63ac4cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:34.589 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.586+0000 7f63ac4cf640 1 -- 172.21.15.3:0/3986143710 >> v1:172.21.15.16:6789/0 conn(0x7f63a40b4e70 legacy=0x7f63a413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:34.589 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.586+0000 7f63ac4cf640 1 -- 172.21.15.3:0/3986143710 shutdown_connections 2024-03-25T18:36:34.589 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.586+0000 7f63ac4cf640 1 -- 172.21.15.3:0/3986143710 >> 172.21.15.3:0/3986143710 conn(0x7f63a40617a0 msgr2=0x7f63a4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:34.589 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.587+0000 7f63ac4cf640 1 -- 172.21.15.3:0/3986143710 shutdown_connections 2024-03-25T18:36:34.590 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.587+0000 7f63ac4cf640 1 -- 172.21.15.3:0/3986143710 wait complete. 2024-03-25T18:36:34.590 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.587+0000 7f63ac4cf640 10 monclient: build_initial_monmap 2024-03-25T18:36:34.590 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.587+0000 7f63ac4cf640 10 monclient: monmap: 2024-03-25T18:36:34.590 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-25T18:36:34.590 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:34.590 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-25T18:36:34.590 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-25T18:36:34.590 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:34.590 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-25T18:36:34.590 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-25T18:36:34.590 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.noname-b 2024-03-25T18:36:34.590 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-25T18:36:34.590 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:36:34.590 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.588+0000 7f63ac4cf640 1 librados: starting msgr at 2024-03-25T18:36:34.591 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.588+0000 7f63ac4cf640 1 librados: starting objecter 2024-03-25T18:36:34.591 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.588+0000 7f63ac4cf640 1 Processor -- start 2024-03-25T18:36:34.591 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.588+0000 7f63ac4cf640 1 -- start start 2024-03-25T18:36:34.591 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.588+0000 7f63ac4cf640 1 librados: setting wanted keys 2024-03-25T18:36:34.591 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.588+0000 7f63ac4cf640 1 librados: calling monclient init 2024-03-25T18:36:34.591 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.588+0000 7f63ac4cf640 10 monclient: init 2024-03-25T18:36:34.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.589+0000 7f63ac4cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:34.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.589+0000 7f63ac4cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:36:34.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.589+0000 7f63ac4cf640 10 monclient(hunting): picked mon.noname-c con 0x7f63a40b1b00 addr v1:172.21.15.3:6790/0 2024-03-25T18:36:34.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.589+0000 7f63ac4cf640 10 monclient(hunting): picked mon.noname-b con 0x7f63a40b44b0 addr v1:172.21.15.16:6789/0 2024-03-25T18:36:34.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.589+0000 7f63ac4cf640 10 monclient(hunting): picked mon.noname-a con 0x7f63a40b4e70 addr v1:172.21.15.3:6789/0 2024-03-25T18:36:34.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.589+0000 7f63ac4cf640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a413a750 con 0x7f63a40b4e70 2024-03-25T18:36:34.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.589+0000 7f63ac4cf640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a413c150 con 0x7f63a40b44b0 2024-03-25T18:36:34.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.589+0000 7f63ac4cf640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a413c430 con 0x7f63a40b1b00 2024-03-25T18:36:34.593 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.589+0000 7f63ac4cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:34.593 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.589+0000 7f63ac4cf640 10 monclient(hunting): authenticate will time out at 8048.849913s 2024-03-25T18:36:34.603 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.602+0000 7f63a9a6a640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f63a40b44b0 0x7f63a4144f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.3:55064/0 (socket says 172.21.15.3:55064) 2024-03-25T18:36:34.603 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.602+0000 7f63a9a6a640 1 -- 172.21.15.3:0/2144469771 learned_addr learned my addr 172.21.15.3:0/2144469771 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-25T18:36:34.604 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.604+0000 7f639affd640 10 client.?.objecter ms_handle_connect 0x7f63a40b44b0 2024-03-25T18:36:34.604 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.604+0000 7f639affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:34.609 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.608+0000 7f639affd640 10 client.?.objecter ms_handle_connect 0x7f63a40b1b00 2024-03-25T18:36:34.609 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.608+0000 7f639affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:34.617 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.617+0000 7f639affd640 10 client.?.objecter ms_handle_connect 0x7f63a40b4e70 2024-03-25T18:36:34.617 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.617+0000 7f639affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:34.623 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.623+0000 7f639affd640 1 -- 172.21.15.3:0/2144469771 <== mon.1 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1492982711 0 0) 0x7f63a413c150 con 0x7f63a40b44b0 2024-03-25T18:36:34.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.623+0000 7f639affd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:34.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.623+0000 7f639affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:34.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.623+0000 7f639affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:34.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.623+0000 7f639affd640 10 monclient(hunting): my global_id is 4693 2024-03-25T18:36:34.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.623+0000 7f639affd640 1 -- 172.21.15.3:0/2144469771 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6388004eb0 con 0x7f63a40b44b0 2024-03-25T18:36:34.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.625+0000 7f639affd640 1 -- 172.21.15.3:0/2144469771 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3175015875 0 0) 0x7f63a413a750 con 0x7f63a40b4e70 2024-03-25T18:36:34.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.625+0000 7f639affd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:34.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.625+0000 7f639affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:34.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.625+0000 7f639affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:34.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.625+0000 7f639affd640 10 monclient(hunting): my global_id is 4755 2024-03-25T18:36:34.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.625+0000 7f639affd640 1 -- 172.21.15.3:0/2144469771 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63a413c150 con 0x7f63a40b4e70 2024-03-25T18:36:34.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.628+0000 7f639affd640 1 -- 172.21.15.3:0/2144469771 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4026539204 0 0) 0x7f63a413c430 con 0x7f63a40b1b00 2024-03-25T18:36:34.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.628+0000 7f639affd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:34.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.628+0000 7f639affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:34.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.628+0000 7f639affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:34.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.628+0000 7f639affd640 10 monclient(hunting): my global_id is 4697 2024-03-25T18:36:34.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.628+0000 7f639affd640 1 -- 172.21.15.3:0/2144469771 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63a413a750 con 0x7f63a40b1b00 2024-03-25T18:36:34.641 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.640+0000 7f639affd640 1 -- 172.21.15.3:0/2144469771 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 977863145 0 0) 0x7f63a413c150 con 0x7f63a40b4e70 2024-03-25T18:36:34.641 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.640+0000 7f639affd640 1 -- 172.21.15.3:0/2144469771 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63a413c430 con 0x7f63a40b4e70 2024-03-25T18:36:34.641 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.640+0000 7f639affd640 1 -- 172.21.15.3:0/2144469771 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f639c001cb0 con 0x7f63a40b4e70 2024-03-25T18:36:34.641 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.640+0000 7f639affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:34.641 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.640+0000 7f639affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:36:34.641 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.640+0000 7f639affd640 10 monclient(hunting): dump: 2024-03-25T18:36:34.641 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:36:34.642 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:36:34.642 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:34.642 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:34.642 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:34.642 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:36:34.642 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:36:34.642 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:36:34.642 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:36:34.642 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:36:34.642 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.640+0000 7f639affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:34.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.650+0000 7f639affd640 1 -- 172.21.15.3:0/2144469771 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2952652802 0 0) 0x7f63a413c430 con 0x7f63a40b4e70 2024-03-25T18:36:34.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.650+0000 7f639affd640 1 -- 172.21.15.3:0/2144469771 >> v1:172.21.15.3:6790/0 conn(0x7f63a40b1b00 legacy=0x7f63a413aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:34.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.650+0000 7f639affd640 1 -- 172.21.15.3:0/2144469771 >> v1:172.21.15.16:6789/0 conn(0x7f63a40b44b0 legacy=0x7f63a4144f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:34.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.650+0000 7f639affd640 10 monclient: _finish_hunting 0 2024-03-25T18:36:34.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.650+0000 7f639affd640 1 monclient: found mon.a 2024-03-25T18:36:34.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.650+0000 7f639affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:34.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.650+0000 7f639affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:36:34.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.650+0000 7f639affd640 1 -- 172.21.15.3:0/2144469771 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a413c710 con 0x7f63a40b4e70 2024-03-25T18:36:34.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.650+0000 7f639affd640 10 monclient: _finish_auth 0 2024-03-25T18:36:34.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.650+0000 7f639affd640 10 monclient: _check_auth_tickets 2024-03-25T18:36:34.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.650+0000 7f639affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:34.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.650+0000 7f63ac4cf640 5 monclient: authenticate success, global_id 4755 2024-03-25T18:36:34.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.650+0000 7f63ac4cf640 10 monclient: _renew_subs 2024-03-25T18:36:34.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.650+0000 7f63ac4cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:36:34.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.650+0000 7f63ac4cf640 1 -- 172.21.15.3:0/2144469771 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63a413c9f0 con 0x7f63a40b4e70 2024-03-25T18:36:34.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.650+0000 7f63ac4cf640 10 client.4755.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:34.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.650+0000 7f63ac4cf640 10 monclient: _renew_subs 2024-03-25T18:36:34.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.650+0000 7f63ac4cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:36:34.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.650+0000 7f63ac4cf640 1 -- 172.21.15.3:0/2144469771 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63a4143530 con 0x7f63a40b4e70 2024-03-25T18:36:34.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.650+0000 7f63ac4cf640 1 librados: init done 2024-03-25T18:36:34.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.651+0000 7f63ac4cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:34.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.651+0000 7f63ac4cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:34.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.651+0000 7f63ac4cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:36:34.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.651+0000 7f63ac4cf640 1 -- 172.21.15.3:0/2144469771 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63a413a900 con 0x7f63a40b4e70 2024-03-25T18:36:34.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.962+0000 7f639affd640 1 -- 172.21.15.3:0/2144469771 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f639c003990 con 0x7f63a40b4e70 2024-03-25T18:36:34.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.962+0000 7f639affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:34.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.962+0000 7f639affd640 1 -- 172.21.15.3:0/2144469771 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3357630750 0 0) 0x7f639c002100 con 0x7f63a40b4e70 2024-03-25T18:36:34.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.962+0000 7f639affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:34.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.962+0000 7f639affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:34.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.962+0000 7f639affd640 10 monclient: dump: 2024-03-25T18:36:34.963 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-25T18:36:34.963 INFO:teuthology.misc.health.smithi003.stderr:fsid da0b8676-a454-4df7-b780-787f16fb595c 2024-03-25T18:36:34.963 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:34.963 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-25T18:24:15.374468+0000 2024-03-25T18:36:34.964 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:34.964 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 3 2024-03-25T18:36:34.964 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-25T18:36:34.964 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.16:6789/0 mon.b 2024-03-25T18:36:34.964 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-25T18:36:34.964 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-25T18:36:34.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.962+0000 7f63a8a68640 4 set_mon_vals no callback set 2024-03-25T18:36:34.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.963+0000 7f639affd640 1 -- 172.21.15.3:0/2144469771 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4287198043 0 0) 0x7f639c019bd0 con 0x7f63a40b4e70 2024-03-25T18:36:34.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.964+0000 7f63aa26b640 10 monclient: get_auth_request con 0x7f638806de20 auth_method 0 2024-03-25T18:36:34.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.964+0000 7f639affd640 10 client.4755.objecter ms_handle_connect 0x7f638806de20 2024-03-25T18:36:34.969 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.968+0000 7f639affd640 1 -- 172.21.15.3:0/2144469771 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2800263165 0 0) 0x7f639c053830 con 0x7f63a40b4e70 2024-03-25T18:36:34.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.968+0000 7f639affd640 10 client.4755.objecter ms_dispatch 0x7f63a4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:36:34.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.968+0000 7f639affd640 3 client.4755.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:36:34.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.968+0000 7f639affd640 3 client.4755.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:36:34.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.968+0000 7f639affd640 20 client.4755.objecter dump_active .. 0 homeless 2024-03-25T18:36:34.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.968+0000 7f639affd640 1 -- 172.21.15.3:0/2144469771 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f639c04f170 con 0x7f63a40b4e70 2024-03-25T18:36:34.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.968+0000 7f639affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:34.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:34.968+0000 7f639affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:35.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:35.229+0000 7f63ac4cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:35.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:35.229+0000 7f63ac4cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:35.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:35.229+0000 7f63ac4cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-25T18:36:35.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:35.229+0000 7f63ac4cf640 1 -- 172.21.15.3:0/2144469771 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63a404f730 con 0x7f63a40b4e70 2024-03-25T18:36:35.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:35.263+0000 7f639affd640 1 -- 172.21.15.3:0/2144469771 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 716832884) 0x7f639c051270 con 0x7f63a40b4e70 2024-03-25T18:36:35.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:35.263+0000 7f639affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:35.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:35.263+0000 7f639affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:35.264 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-25T18:36:35.264 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":11,"fsid":"da0b8676-a454-4df7-b780-787f16fb595c","created":"2024-03-25T18:25:03.714374+0000","modified":"2024-03-25T18:35:08.249841+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:35:08.249841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8dee66f-c1ad-477e-8d93-f1bf6ec49cac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"41c16611-a0ff-4bd4-8af8-362561ba9102","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3e3857a4-8eff-44cf-b75c-dacf17b4248b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"83f74bdb-fb01-41e8-b3da-f69a7459f916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a765e52f-f1cf-4ad6-8f18-6df9db782869","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9ca8edca-f60e-4490-bc1d-782bc08c5b09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"742ad6e9-6a86-4fb6-bd82-020c991fba92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98deee3b-2dc8-4fbd-b5fb-9a65bc700a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:35.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:35.265+0000 7f63ac4cf640 10 librados: watch_flush enter 2024-03-25T18:36:35.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:35.265+0000 7f63ac4cf640 10 librados: watch_flush exit 2024-03-25T18:36:35.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:35.265+0000 7f63ac4cf640 20 client.4755.objecter shutdown clearing up homeless session... 2024-03-25T18:36:35.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:35.265+0000 7f63ac4cf640 10 client.4755.objecter successfully canceled tick 2024-03-25T18:36:35.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:35.265+0000 7f63ac4cf640 1 -- 172.21.15.3:0/2144469771 >> v1:172.21.15.3:6800/4195140966 conn(0x7f638806de20 legacy=0x7f63880702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:35.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:35.265+0000 7f63ac4cf640 10 monclient: shutdown 2024-03-25T18:36:35.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:35.265+0000 7f63ac4cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:35.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:35.265+0000 7f63ac4cf640 1 -- 172.21.15.3:0/2144469771 >> v1:172.21.15.3:6789/0 conn(0x7f63a40b4e70 legacy=0x7f63a4142e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:35.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:35.265+0000 7f63ac4cf640 1 -- 172.21.15.3:0/2144469771 shutdown_connections 2024-03-25T18:36:35.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:35.265+0000 7f63ac4cf640 1 -- 172.21.15.3:0/2144469771 >> 172.21.15.3:0/2144469771 conn(0x7f63a40617a0 msgr2=0x7f63a40b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:35.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:35.265+0000 7f63ac4cf640 1 -- 172.21.15.3:0/2144469771 shutdown_connections 2024-03-25T18:36:35.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:35.265+0000 7f63ac4cf640 1 -- 172.21.15.3:0/2144469771 wait complete. 2024-03-25T18:36:35.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-25T18:36:35.266+0000 7f63ac4cf640 1 librados: shutdown 2024-03-25T18:36:35.278 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:35.278 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T18:36:35.280 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T18:36:35.280 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T18:36:35.280 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:36:37.156 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:11:27.723 35478== 2024-03-25T18:36:37.156 INFO:tasks.ceph.osd.0.smithi003.stderr:==00:00:11:27.723 35478== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:36:37.199 INFO:tasks.ceph.osd.0.smithi003.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:36:41.381 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:36:41.382 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T18:36:41.382 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T18:36:41.383 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T18:36:41.383 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:36:43.388 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:11:33.953 35484== 2024-03-25T18:36:43.388 INFO:tasks.ceph.osd.1.smithi003.stderr:==00:00:11:33.953 35484== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:36:43.437 INFO:tasks.ceph.osd.1.smithi003.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:36:43.646 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:11:57.375 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:43.707 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:11:57.436 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:43.734 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:11:57.463 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:43.748 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:11:57.478 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:43.758 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:11:57.451 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:43.762 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:11:57.491 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:43.764 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:11:57.457 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:43.790 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:11:57.519 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:43.797 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:11:57.489 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:47.484 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:36:47.484 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T18:36:47.485 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T18:36:47.485 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T18:36:47.485 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:36:49.343 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:11:39.902 35485== 2024-03-25T18:36:49.343 INFO:tasks.ceph.osd.2.smithi003.stderr:==00:00:11:39.902 35485== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:36:49.381 INFO:tasks.ceph.osd.2.smithi003.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:36:53.587 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:36:53.588 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T18:36:53.588 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T18:36:53.588 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T18:36:53.588 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:36:55.443 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:11:45.984 35490== 2024-03-25T18:36:55.443 INFO:tasks.ceph.osd.3.smithi003.stderr:==00:00:11:45.984 35490== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:36:55.486 INFO:tasks.ceph.osd.3.smithi003.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:36:59.690 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:36:59.690 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T18:36:59.691 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T18:36:59.691 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T18:36:59.691 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:37:01.610 INFO:tasks.ceph.osd.4.smithi016.stderr:==00:00:11:52.117 35190== 2024-03-25T18:37:01.610 INFO:tasks.ceph.osd.4.smithi016.stderr:==00:00:11:52.118 35190== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:37:01.646 INFO:tasks.ceph.osd.4.smithi016.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:37:03.676 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:12:17.405 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:03.725 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:12:17.454 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:03.764 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:12:17.493 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:03.778 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:12:17.471 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:03.784 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:12:17.513 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:03.784 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:12:17.477 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:03.787 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:12:17.516 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:03.807 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:12:17.500 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:03.810 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:12:17.539 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:05.793 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:37:05.793 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T18:37:05.794 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T18:37:05.794 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T18:37:05.794 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:37:07.778 INFO:tasks.ceph.osd.5.smithi016.stderr:==00:00:11:58.306 35187== 2024-03-25T18:37:07.778 INFO:tasks.ceph.osd.5.smithi016.stderr:==00:00:11:58.306 35187== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:37:07.828 INFO:tasks.ceph.osd.5.smithi016.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:37:11.895 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:37:11.896 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T18:37:11.896 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T18:37:11.896 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T18:37:11.897 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:37:13.612 INFO:tasks.ceph.osd.6.smithi016.stderr:==00:00:12:04.141 35188== 2024-03-25T18:37:13.612 INFO:tasks.ceph.osd.6.smithi016.stderr:==00:00:12:04.141 35188== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:37:13.663 INFO:tasks.ceph.osd.6.smithi016.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:37:17.998 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:37:17.998 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T18:37:17.999 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T18:37:17.999 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T18:37:17.999 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:37:19.487 INFO:tasks.ceph.osd.7.smithi016.stderr:==00:00:12:10.012 35189== 2024-03-25T18:37:19.487 INFO:tasks.ceph.osd.7.smithi016.stderr:==00:00:12:10.012 35189== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:37:19.516 INFO:tasks.ceph.osd.7.smithi016.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:37:23.739 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:12:37.468 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:23.788 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:12:37.517 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:23.801 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:12:37.531 35175== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:23.838 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:12:37.530 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:23.838 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:12:37.568 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:23.840 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:12:37.570 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:23.841 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:12:37.533 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:23.853 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:12:37.545 34644== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:23.854 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:12:37.583 35177== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:24.101 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:37:24.102 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T18:37:24.102 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T18:37:24.102 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T18:37:24.103 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T18:37:24.103 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:37:24.104 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-25T18:37:24.103+0000 7f0c82a00640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i y (PID: 35182) UID: 0 2024-03-25T18:37:24.104 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-25T18:37:24.103+0000 7f0c82a00640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-25T18:37:24.127 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T18:37:24.128 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T18:37:24.128 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:37:24.151 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T18:37:24.151 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T18:37:24.151 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T18:37:24.151 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:37:24.184 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-03-25T18:37:24.171+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35142) UID: 0 2024-03-25T18:37:24.184 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-03-25T18:37:24.173+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T18:37:24.665 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:12:38.394 35175== 2024-03-25T18:37:24.665 INFO:tasks.ceph.mon.a.smithi003.stderr:==00:00:12:38.394 35175== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:37:24.675 INFO:tasks.ceph.mon.a.smithi003.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:37:30.253 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:37:30.253 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T18:37:30.253 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T18:37:30.254 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T18:37:30.254 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:37:30.270 INFO:tasks.ceph.mon.c.smithi003.stderr:2024-03-25T18:37:30.257+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35144) UID: 0 2024-03-25T18:37:30.271 INFO:tasks.ceph.mon.c.smithi003.stderr:2024-03-25T18:37:30.259+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T18:37:30.819 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:12:44.548 35177== 2024-03-25T18:37:30.819 INFO:tasks.ceph.mon.c.smithi003.stderr:==00:00:12:44.548 35177== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:37:30.828 INFO:tasks.ceph.mon.c.smithi003.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:37:36.356 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:37:36.356 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T18:37:36.357 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T18:37:36.357 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T18:37:36.357 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:37:36.374 INFO:tasks.ceph.mon.b.smithi016.stderr:2024-03-25T18:37:36.361+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34640) UID: 0 2024-03-25T18:37:36.374 INFO:tasks.ceph.mon.b.smithi016.stderr:2024-03-25T18:37:36.363+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T18:37:36.894 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:12:50.586 34644== 2024-03-25T18:37:36.894 INFO:tasks.ceph.mon.b.smithi016.stderr:==00:00:12:50.586 34644== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:37:36.904 INFO:tasks.ceph.mon.b.smithi016.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:37:42.459 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:37:42.460 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T18:37:42.460 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T18:37:42.460 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T18:37:42.461 DEBUG:teuthology.orchestra.run.smithi003:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T18:37:42.505 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi003.front.sepia.ceph.com 2024-03-25T18:37:42.505 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T18:37:42.666 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi003.front.sepia.ceph.com 2024-03-25T18:37:42.666 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T18:37:42.758 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi003.front.sepia.ceph.com 2024-03-25T18:37:42.758 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T18:37:42.826 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi003.front.sepia.ceph.com 2024-03-25T18:37:42.826 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T18:37:42.907 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi016.front.sepia.ceph.com 2024-03-25T18:37:42.907 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T18:37:43.029 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi016.front.sepia.ceph.com 2024-03-25T18:37:43.029 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T18:37:43.110 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi016.front.sepia.ceph.com 2024-03-25T18:37:43.110 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T18:37:43.176 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi016.front.sepia.ceph.com 2024-03-25T18:37:43.176 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T18:37:43.233 INFO:tasks.ceph:Archiving mon data... 2024-03-25T18:37:43.234 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621281/data/mon.a.tgz 2024-03-25T18:37:43.235 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2024-03-25T18:37:43.251 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.O9oU8qwfsE 2024-03-25T18:37:43.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.O9oU8qwfsE 2024-03-25T18:37:43.354 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.O9oU8qwfsE 2024-03-25T18:37:43.440 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.O9oU8qwfsE is 133KB 2024-03-25T18:37:43.492 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.O9oU8qwfsE 2024-03-25T18:37:43.506 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621281/data/mon.c.tgz 2024-03-25T18:37:43.507 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2024-03-25T18:37:43.560 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.MLMv21MWqy 2024-03-25T18:37:43.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.MLMv21MWqy 2024-03-25T18:37:43.648 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.MLMv21MWqy 2024-03-25T18:37:43.732 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.MLMv21MWqy is 133KB 2024-03-25T18:37:43.782 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.MLMv21MWqy 2024-03-25T18:37:43.796 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621281/data/mon.b.tgz 2024-03-25T18:37:43.796 DEBUG:teuthology.orchestra.run.smithi016:> mktemp 2024-03-25T18:37:43.813 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.KqGxcqgGah 2024-03-25T18:37:43.814 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.KqGxcqgGah 2024-03-25T18:37:43.922 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.KqGxcqgGah 2024-03-25T18:37:44.007 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.KqGxcqgGah is 133KB 2024-03-25T18:37:44.028 DEBUG:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.KqGxcqgGah 2024-03-25T18:37:44.041 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T18:37:44.042 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:37:44.044 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:37:44.340 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:37:44.518 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T18:37:44.520 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621281/remote/smithi003/crash 2024-03-25T18:37:44.521 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:37:44.548 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621281/remote/smithi016/crash 2024-03-25T18:37:44.549 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:37:44.577 INFO:tasks.ceph:Compressing logs... 2024-03-25T18:37:44.577 DEBUG:teuthology.orchestra.run.smithi003:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:37:44.593 DEBUG:teuthology.orchestra.run.smithi016:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:37:44.664 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T18:37:44.664 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35039.log 2024-03-25T18:37:44.664 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T18:37:44.664 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.35039.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T18:37:44.664 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- 86.9% /var/log/ceph/valgrind/osd.6.log 2024-03-25T18:37:44.664 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35039.log.gz 2024-03-25T18:37:44.664 INFO:teuthology.orchestra.run.smithi016.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T18:37:44.665 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T18:37:44.665 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/valgrind/osd.6.log: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T18:37:44.665 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T18:37:44.665 INFO:teuthology.orchestra.run.smithi016.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T18:37:44.665 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34957.log 2024-03-25T18:37:44.665 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35088.log 2024-03-25T18:37:44.665 INFO:teuthology.orchestra.run.smithi016.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T18:37:44.665 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/valgrind/osd.4.log: /var/log/ceph/ceph-client.admin.34957.log: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T18:37:44.665 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34797.log 2024-03-25T18:37:44.665 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:37:44.665 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35088.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34957.log.gz 2024-03-25T18:37:44.665 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34917.log 2024-03-25T18:37:44.665 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph.audit.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35088.log.gz 2024-03-25T18:37:44.666 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T18:37:44.666 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34797.log.gz 2024-03-25T18:37:44.666 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34917.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T18:37:44.666 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:37:44.666 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T18:37:44.666 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34917.log.gz 2024-03-25T18:37:44.666 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T18:37:44.666 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34877.log 2024-03-25T18:37:44.666 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34837.log 2024-03-25T18:37:44.666 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34877.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:37:44.666 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34837.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34877.log.gz 2024-03-25T18:37:44.666 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34997.log 2024-03-25T18:37:44.666 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.34837.log.gz 2024-03-25T18:37:44.666 INFO:teuthology.orchestra.run.smithi016.stderr: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:37:44.666 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T18:37:44.667 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34997.log: /var/log/ceph/ceph-mon.b.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34997.log.gz 2024-03-25T18:37:44.667 INFO:teuthology.orchestra.run.smithi016.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T18:37:44.670 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35591.log 2024-03-25T18:37:44.671 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39220.log 2024-03-25T18:37:44.671 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.35591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41956.log 2024-03-25T18:37:44.671 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39220.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T18:37:44.671 INFO:teuthology.orchestra.run.smithi003.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.35591.log.gz 2024-03-25T18:37:44.671 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40195.log 2024-03-25T18:37:44.671 INFO:teuthology.orchestra.run.smithi003.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39220.log.gz 2024-03-25T18:37:44.671 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41058.log 2024-03-25T18:37:44.671 INFO:teuthology.orchestra.run.smithi003.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41956.log.gz 2024-03-25T18:37:44.671 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39298.log 2024-03-25T18:37:44.671 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41410.log 2024-03-25T18:37:44.671 INFO:teuthology.orchestra.run.smithi003.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40195.log.gz 2024-03-25T18:37:44.671 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39337.log 2024-03-25T18:37:44.671 INFO:teuthology.orchestra.run.smithi003.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41058.log.gz 2024-03-25T18:37:44.671 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41410.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T18:37:44.671 INFO:teuthology.orchestra.run.smithi003.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39298.log.gz 2024-03-25T18:37:44.672 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39337.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41410.log.gz 2024-03-25T18:37:44.672 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T18:37:44.672 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T18:37:44.672 INFO:teuthology.orchestra.run.smithi003.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39337.log.gz 2024-03-25T18:37:44.672 INFO:teuthology.orchestra.run.smithi003.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T18:37:44.672 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T18:37:44.672 INFO:teuthology.orchestra.run.smithi003.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T18:37:44.672 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T18:37:44.672 INFO:teuthology.orchestra.run.smithi003.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T18:37:44.672 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T18:37:44.672 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/valgrind/mon.a.log: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T18:37:44.672 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42151.log 2024-03-25T18:37:44.672 INFO:teuthology.orchestra.run.smithi003.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T18:37:44.672 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41371.log 2024-03-25T18:37:44.672 INFO:teuthology.orchestra.run.smithi003.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T18:37:44.673 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.42151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40390.log 2024-03-25T18:37:44.673 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40588.log 2024-03-25T18:37:44.673 INFO:teuthology.orchestra.run.smithi003.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.42151.log.gz 2024-03-25T18:37:44.673 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40390.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41371.log.gz 2024-03-25T18:37:44.673 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39961.log 2024-03-25T18:37:44.673 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41800.log 2024-03-25T18:37:44.673 INFO:teuthology.orchestra.run.smithi003.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40390.log.gz 2024-03-25T18:37:44.673 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40862.log 2024-03-25T18:37:44.673 INFO:teuthology.orchestra.run.smithi003.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40588.log.gz 2024-03-25T18:37:44.673 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41292.log 2024-03-25T18:37:44.673 INFO:teuthology.orchestra.run.smithi003.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39961.log.gz 2024-03-25T18:37:44.673 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42346.log 2024-03-25T18:37:44.673 INFO:teuthology.orchestra.run.smithi003.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41800.log.gz 2024-03-25T18:37:44.673 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41292.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40862.log.gz 2024-03-25T18:37:44.673 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39883.log 2024-03-25T18:37:44.674 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.42346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41839.log 2024-03-25T18:37:44.674 INFO:teuthology.orchestra.run.smithi003.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41292.log.gz 2024-03-25T18:37:44.674 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40000.log 2024-03-25T18:37:44.674 INFO:teuthology.orchestra.run.smithi003.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.42346.log.gz 2024-03-25T18:37:44.674 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41839.log: gzip 86.3% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.39883.log.gz -- 2024-03-25T18:37:44.674 INFO:teuthology.orchestra.run.smithi003.stderr: /var/log/ceph/ceph-client.admin.42307.log 2024-03-25T18:37:44.674 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41878.log 2024-03-25T18:37:44.674 INFO:teuthology.orchestra.run.smithi003.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41839.log.gz 2024-03-25T18:37:44.674 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.42307.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40000.log.gz 2024-03-25T18:37:44.674 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29318.log 2024-03-25T18:37:44.674 INFO:teuthology.orchestra.run.smithi003.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.42307.log.gz 2024-03-25T18:37:44.674 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41136.log 2024-03-25T18:37:44.674 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph.tmp-client.admin.29318.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph.tmp-client.admin.29318.log.gz /var/log/ceph/ceph-client.admin.40273.log 2024-03-25T18:37:44.674 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-25T18:37:44.674 INFO:teuthology.orchestra.run.smithi003.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41878.log.gz 2024-03-25T18:37:44.675 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40549.log 2024-03-25T18:37:44.675 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41097.log 2024-03-25T18:37:44.675 INFO:teuthology.orchestra.run.smithi003.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41136.log.gz 2024-03-25T18:37:44.675 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40549.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:37:44.675 INFO:teuthology.orchestra.run.smithi003.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40273.log.gz 2024-03-25T18:37:44.675 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42385.log 2024-03-25T18:37:44.675 INFO:teuthology.orchestra.run.smithi003.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40549.log.gz 2024-03-25T18:37:44.675 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph.audit.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41097.log.gz 2024-03-25T18:37:44.675 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36601.log 2024-03-25T18:37:44.675 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.42385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39688.log 2024-03-25T18:37:44.675 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36601.log: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:37:44.675 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40312.log 2024-03-25T18:37:44.675 INFO:teuthology.orchestra.run.smithi003.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.42385.log.gz 2024-03-25T18:37:44.675 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39688.log: gzip 87.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.36601.log.gz 2024-03-25T18:37:44.675 INFO:teuthology.orchestra.run.smithi003.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41683.log 2024-03-25T18:37:44.675 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40312.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39688.log.gz 2024-03-25T18:37:44.676 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41253.log 2024-03-25T18:37:44.676 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41527.log 2024-03-25T18:37:44.676 INFO:teuthology.orchestra.run.smithi003.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40312.log.gz 2024-03-25T18:37:44.676 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41253.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41683.log.gz 2024-03-25T18:37:44.676 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41019.log 2024-03-25T18:37:44.676 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41175.log 2024-03-25T18:37:44.676 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41527.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41253.log.gz 2024-03-25T18:37:44.676 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40156.log 2024-03-25T18:37:44.676 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41175.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41527.log.gz 2024-03-25T18:37:44.676 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T18:37:44.676 INFO:teuthology.orchestra.run.smithi003.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41019.log.gz 2024-03-25T18:37:44.676 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39805.log 2024-03-25T18:37:44.676 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40156.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41175.log.gz 2024-03-25T18:37:44.676 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41917.log 2024-03-25T18:37:44.676 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39805.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40156.log.gz 2024-03-25T18:37:44.676 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37607.log 2024-03-25T18:37:44.677 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39180.log 2024-03-25T18:37:44.677 INFO:teuthology.orchestra.run.smithi003.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39805.log.gz 2024-03-25T18:37:44.677 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37607.log: 86.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.41917.log.gz 2024-03-25T18:37:44.677 INFO:teuthology.orchestra.run.smithi003.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41761.log 2024-03-25T18:37:44.677 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35721.log 2024-03-25T18:37:44.677 INFO:teuthology.orchestra.run.smithi003.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37607.log.gz 2024-03-25T18:37:44.677 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39571.log 2024-03-25T18:37:44.677 INFO:teuthology.orchestra.run.smithi003.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39180.log.gz 2024-03-25T18:37:44.677 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.35721.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41761.log.gz 2024-03-25T18:37:44.677 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41605.log 2024-03-25T18:37:44.677 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40744.log 2024-03-25T18:37:44.677 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39571.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.35721.log.gz 2024-03-25T18:37:44.677 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39100.log 2024-03-25T18:37:44.677 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40744.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39571.log.gz 2024-03-25T18:37:44.678 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40510.log 2024-03-25T18:37:44.678 INFO:teuthology.orchestra.run.smithi003.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41605.log.gz 2024-03-25T18:37:44.678 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39100.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 85.8% 2024-03-25T18:37:44.678 INFO:teuthology.orchestra.run.smithi003.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40744.log.gz 2024-03-25T18:37:44.678 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40078.log 2024-03-25T18:37:44.678 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-osd.3.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39100.log.gz 2024-03-25T18:37:44.678 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39259.log 2024-03-25T18:37:44.678 INFO:teuthology.orchestra.run.smithi003.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40510.log.gz 2024-03-25T18:37:44.678 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41722.log 2024-03-25T18:37:44.678 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39259.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40078.log.gz 2024-03-25T18:37:44.678 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40429.log 2024-03-25T18:37:44.678 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41995.log 2024-03-25T18:37:44.678 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41722.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39259.log.gz 2024-03-25T18:37:44.679 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41488.log 2024-03-25T18:37:44.679 INFO:teuthology.orchestra.run.smithi003.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41722.log.gz 2024-03-25T18:37:44.679 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40941.log 2024-03-25T18:37:44.679 INFO:teuthology.orchestra.run.smithi003.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40429.log.gz 2024-03-25T18:37:44.679 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41488.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41995.log.gz 2024-03-25T18:37:44.679 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39493.log 2024-03-25T18:37:44.679 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose 85.8% -- /var/log/ceph/ceph-client.admin.41214.log -- replaced with /var/log/ceph/ceph-client.admin.41488.log.gz 2024-03-25T18:37:44.679 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-25T18:37:44.679 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40941.log: /var/log/ceph/ceph-client.admin.39493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42073.log 2024-03-25T18:37:44.679 INFO:teuthology.orchestra.run.smithi003.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40941.log.gz 2024-03-25T18:37:44.679 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39376.log 2024-03-25T18:37:44.679 INFO:teuthology.orchestra.run.smithi003.stderr: 86.9%/var/log/ceph/ceph-client.admin.42073.log: -- replaced with /var/log/ceph/ceph-client.admin.39493.log.gz 2024-03-25T18:37:44.679 INFO:teuthology.orchestra.run.smithi003.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41214.log.gz 2024-03-25T18:37:44.679 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41449.log 2024-03-25T18:37:44.680 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39139.log 2024-03-25T18:37:44.680 INFO:teuthology.orchestra.run.smithi003.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42073.log.gz 2024-03-25T18:37:44.680 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39376.log: /var/log/ceph/ceph-client.admin.41449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38867.log 2024-03-25T18:37:44.680 INFO:teuthology.orchestra.run.smithi003.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39376.log.gz 2024-03-25T18:37:44.680 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39139.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41449.log.gz 2024-03-25T18:37:44.680 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T18:37:44.680 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39766.log 2024-03-25T18:37:44.680 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38867.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39139.log.gz 2024-03-25T18:37:44.680 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40666.log 2024-03-25T18:37:44.680 INFO:teuthology.orchestra.run.smithi003.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38867.log.gz 2024-03-25T18:37:44.680 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39610.log 2024-03-25T18:37:44.680 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40666.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39766.log.gz 2024-03-25T18:37:44.680 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39649.log 2024-03-25T18:37:44.680 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39610.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40666.log.gz 2024-03-25T18:37:44.680 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40117.log 2024-03-25T18:37:44.681 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39649.log: gzip -5 --verbose 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39610.log.gz 2024-03-25T18:37:44.681 INFO:teuthology.orchestra.run.smithi003.stderr: -- /var/log/ceph/ceph-client.admin.40234.log 2024-03-25T18:37:44.681 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40117.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39649.log.gz 2024-03-25T18:37:44.681 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42112.log 2024-03-25T18:37:44.681 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40234.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40117.log.gz 2024-03-25T18:37:44.681 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40823.log 2024-03-25T18:37:44.681 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.42112.log: 85.9%gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T18:37:44.681 INFO:teuthology.orchestra.run.smithi003.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40234.log.gz 2024-03-25T18:37:44.681 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39532.log 2024-03-25T18:37:44.681 INFO:teuthology.orchestra.run.smithi003.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.42112.log.gz 2024-03-25T18:37:44.681 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40039.log 2024-03-25T18:37:44.681 INFO:teuthology.orchestra.run.smithi003.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40823.log.gz 2024-03-25T18:37:44.681 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39727.log 2024-03-25T18:37:44.681 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39532.log: /var/log/ceph/ceph-client.admin.40039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39844.log 2024-03-25T18:37:44.682 INFO:teuthology.orchestra.run.smithi003.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39532.log.gz/var/log/ceph/ceph-client.admin.39727.log: 2024-03-25T18:37:44.682 INFO:teuthology.orchestra.run.smithi003.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40039.log.gz 2024-03-25T18:37:44.682 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41566.log 2024-03-25T18:37:44.682 INFO:teuthology.orchestra.run.smithi003.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39727.log.gz 2024-03-25T18:37:44.682 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39844.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39844.log.gz 2024-03-25T18:37:44.682 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40351.log 2024-03-25T18:37:44.682 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42229.log 2024-03-25T18:37:44.682 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35184.log 2024-03-25T18:37:44.682 INFO:teuthology.orchestra.run.smithi003.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41566.log.gz 2024-03-25T18:37:44.682 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.42229.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40351.log.gz 2024-03-25T18:37:44.682 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42229.log.gz 2024-03-25T18:37:44.682 INFO:teuthology.orchestra.run.smithi003.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39454.log 2024-03-25T18:37:44.682 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.35184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42268.log 2024-03-25T18:37:44.682 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39454.log: gzip -5 90.1% --verbose -- /var/log/ceph/ceph-client.admin.35489.log 2024-03-25T18:37:44.683 INFO:teuthology.orchestra.run.smithi003.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35184.log.gz 2024-03-25T18:37:44.683 INFO:teuthology.orchestra.run.smithi003.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39454.log.gz 2024-03-25T18:37:44.683 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.42268.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:37:44.683 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.35489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40705.log 2024-03-25T18:37:44.683 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42268.log.gz 2024-03-25T18:37:44.683 INFO:teuthology.orchestra.run.smithi003.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.35489.log.gz 2024-03-25T18:37:44.683 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42190.log 2024-03-25T18:37:44.683 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T18:37:44.683 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40705.log: /var/log/ceph/ceph-client.admin.42190.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40705.log.gz 2024-03-25T18:37:44.683 INFO:teuthology.orchestra.run.smithi003.stderr: 92.9% -- replaced with /var/log/ceph/ceph.log.gz 86.4% -- replaced with /var/log/ceph/ceph-client.admin.42190.log.gz 2024-03-25T18:37:44.683 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-25T18:37:44.683 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40468.log 2024-03-25T18:37:44.683 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42034.log 2024-03-25T18:37:44.683 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39922.log 2024-03-25T18:37:44.683 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.42034.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40468.log.gz 2024-03-25T18:37:44.684 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41331.log 2024-03-25T18:37:44.684 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39922.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42034.log.gz 2024-03-25T18:37:44.684 INFO:teuthology.orchestra.run.smithi003.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39922.log.gz 2024-03-25T18:37:44.684 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39415.log 2024-03-25T18:37:44.684 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40783.log 2024-03-25T18:37:44.684 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39415.log: /var/log/ceph/ceph-client.admin.41331.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39415.log.gz 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41331.log.gz 2024-03-25T18:37:44.684 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-25T18:37:44.684 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41644.log 2024-03-25T18:37:44.684 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40980.log 2024-03-25T18:37:44.684 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41644.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40783.log.gz 2024-03-25T18:37:44.684 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40627.log 2024-03-25T18:37:44.684 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40980.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41644.log.gz 2024-03-25T18:37:44.684 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40902.log 2024-03-25T18:37:44.685 INFO:teuthology.orchestra.run.smithi003.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40980.log.gz 2024-03-25T18:37:44.685 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40627.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40627.log.gz 2024-03-25T18:37:44.685 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T18:37:44.685 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40902.log: /var/log/ceph/ceph-mon.c.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40902.log.gz 2024-03-25T18:37:44.717 INFO:teuthology.orchestra.run.smithi003.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T18:37:45.463 INFO:teuthology.orchestra.run.smithi003.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T18:37:45.521 INFO:teuthology.orchestra.run.smithi016.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T18:37:45.720 INFO:teuthology.orchestra.run.smithi016.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T18:37:45.758 INFO:teuthology.orchestra.run.smithi003.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T18:37:45.817 INFO:teuthology.orchestra.run.smithi016.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T18:37:45.835 INFO:teuthology.orchestra.run.smithi003.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T18:37:45.853 INFO:teuthology.orchestra.run.smithi003.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T18:37:45.855 INFO:teuthology.orchestra.run.smithi016.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T18:37:49.669 INFO:teuthology.orchestra.run.smithi003.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T18:37:49.724 INFO:teuthology.orchestra.run.smithi003.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T18:37:49.726 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-25T18:37:49.726 INFO:teuthology.orchestra.run.smithi003.stderr:real 0m5.121s 2024-03-25T18:37:49.726 INFO:teuthology.orchestra.run.smithi003.stderr:user 0m14.159s 2024-03-25T18:37:49.726 INFO:teuthology.orchestra.run.smithi003.stderr:sys 0m0.504s 2024-03-25T18:37:49.921 INFO:teuthology.orchestra.run.smithi016.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T18:37:49.923 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-25T18:37:49.923 INFO:teuthology.orchestra.run.smithi016.stderr:real 0m5.289s 2024-03-25T18:37:49.923 INFO:teuthology.orchestra.run.smithi016.stderr:user 0m9.322s 2024-03-25T18:37:49.923 INFO:teuthology.orchestra.run.smithi016.stderr:sys 0m0.296s 2024-03-25T18:37:49.923 INFO:tasks.ceph:Archiving logs... 2024-03-25T18:37:49.923 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621281/remote/smithi003/log 2024-03-25T18:37:49.925 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:37:52.182 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621281/remote/smithi016/log 2024-03-25T18:37:52.183 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:37:53.638 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:37:53.832 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=9836c80bc66c4df68cdcb7e8f2daeb32 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:37:53.836 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T18:37:53.845 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T18:37:53.872 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T18:37:53.873 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:37:53.886 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:37:53.962 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T18:37:53.970 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T18:37:53.970 DEBUG:teuthology.orchestra.run.smithi003:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:37:53.973 DEBUG:teuthology.orchestra.run.smithi016:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:37:53.986 INFO:teuthology.orchestra.run.smithi003.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:37:53.988 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:37:53.991 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:37:53.992 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2024-03-25T18:37:53.992 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 64 +581us[ +460us] +/- 75ms 2024-03-25T18:37:53.992 INFO:teuthology.orchestra.run.smithi003.stdout:^* hv02.front.sepia.ceph.com 3 8 377 9 -1014us[-1134us] +/- 61ms 2024-03-25T18:37:53.992 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 0 +835us[ +835us] +/- 66ms 2024-03-25T18:37:53.992 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:37:53.992 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:37:53.993 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-03-25T18:37:53.993 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 67 +573us[ +608us] +/- 75ms 2024-03-25T18:37:53.993 INFO:teuthology.orchestra.run.smithi016.stdout:^* hv02.front.sepia.ceph.com 3 6 377 12 -1078us[-1044us] +/- 61ms 2024-03-25T18:37:53.993 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 +777us[ +777us] +/- 66ms 2024-03-25T18:37:53.993 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:37:53.994 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T18:37:54.002 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T18:37:54.003 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T18:37:54.011 DEBUG:teuthology.orchestra.run.smithi003:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:37:54.103 DEBUG:teuthology.orchestra.run.smithi016:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:37:54.174 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:37:54.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:37:54.205 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:37:54.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:37:54.235 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T18:37:54.244 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T18:37:54.252 INFO:teuthology.task.internal:Duration was 7775.374628 seconds 2024-03-25T18:37:54.252 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T18:37:54.260 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T18:37:54.260 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:37:54.262 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:37:54.304 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:37:54.320 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:37:54.672 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T18:37:54.673 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi003.front.sepia.ceph.com 2024-03-25T18:37:54.673 DEBUG:teuthology.orchestra.run.smithi003:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:37:54.733 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi016.front.sepia.ceph.com 2024-03-25T18:37:54.733 DEBUG:teuthology.orchestra.run.smithi016:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:37:54.753 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T18:37:54.753 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:37:54.775 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:37:54.821 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T18:37:54.821 DEBUG:teuthology.orchestra.run.smithi003:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:37:54.845 DEBUG:teuthology.orchestra.run.smithi016:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:37:54.965 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T18:37:54.974 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T18:37:54.975 DEBUG:teuthology.orchestra.run.smithi003:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:37:55.001 DEBUG:teuthology.orchestra.run.smithi016:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:37:55.027 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T18:37:55.036 DEBUG:teuthology.orchestra.run.smithi003:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:37:55.043 DEBUG:teuthology.orchestra.run.smithi016:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:37:55.066 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = core 2024-03-25T18:37:55.093 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = core 2024-03-25T18:37:55.117 DEBUG:teuthology.orchestra.run.smithi003:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:37:55.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:37:55.139 DEBUG:teuthology.orchestra.run.smithi016:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:37:55.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:37:55.171 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T18:37:55.180 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T18:37:55.181 DEBUG:teuthology.misc:Transferring archived files from smithi003:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621281/remote/smithi003 2024-03-25T18:37:55.181 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:37:55.215 DEBUG:teuthology.misc:Transferring archived files from smithi016:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621281/remote/smithi016 2024-03-25T18:37:55.216 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:37:55.250 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T18:37:55.250 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:37:55.253 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:37:55.300 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T18:37:55.309 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T18:37:55.309 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T18:37:55.317 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T18:37:55.317 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:37:55.320 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:37:55.332 INFO:teuthology.orchestra.run.smithi003.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:37 /home/ubuntu/cephtest 2024-03-25T18:37:55.354 INFO:teuthology.orchestra.run.smithi016.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:37 /home/ubuntu/cephtest 2024-03-25T18:37:55.355 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T18:37:55.364 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T18:37:55.383 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T18:37:55.408 DEBUG:teuthology.nuke:shortname: smithi003 2024-03-25T18:37:55.409 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:37:55.434 DEBUG:teuthology.nuke:shortname: smithi016 2024-03-25T18:37:55.434 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:37:55.437 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621281', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:19:58.073540', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGb6LbRgpGpUFYcRAE4NBLu+f/g5s85+pn1GZ0fF1ALVY6cOt3bertjioP6NLkPBZ2n6pGSpneMDUmqNHXIINj0='} 2024-03-25T18:37:55.459 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621281', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:19:58.072166', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDZzY+f/rTpKVXTkcqZ4u8ZEZyO6gAkY6E41GQQjYvdPQO9C1pOLTahnFTuRxn6R+lK4BV9WlwaNA0pFgkOC2Gg='} 2024-03-25T18:37:55.482 INFO:teuthology.orchestra.console.smithi003:Power off 2024-03-25T18:37:55.482 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:37:55.508 DEBUG:teuthology.orchestra.console.smithi003:power off output: Chassis Power Control: Down/Off 2024-03-25T18:37:55.509 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:37:55.523 INFO:teuthology.orchestra.console.smithi016:Power off 2024-03-25T18:37:55.523 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:37:55.534 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2024-03-25T18:37:55.545 DEBUG:teuthology.orchestra.console.smithi016:power off output: Chassis Power Control: Down/Off 2024-03-25T18:37:55.545 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:37:55.571 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is on 2024-03-25T18:37:59.535 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:37:59.573 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:37:59.645 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2024-03-25T18:37:59.681 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is on 2024-03-25T18:38:03.651 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:38:03.682 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:38:03.761 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is off 2024-03-25T18:38:03.792 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is off 2024-03-25T18:38:03.863 INFO:teuthology.orchestra.console.smithi003:Power off completed 2024-03-25T18:38:03.893 INFO:teuthology.orchestra.console.smithi016:Power off completed 2024-03-25T18:38:03.998 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 7775.374627828598 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=9836c80bc66c4df68cdcb7e8f2daeb32 status: fail success: false 2024-03-25T18:38:03.998 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:38:04.078 INFO:teuthology.run:FAIL